PredictLM v11.0 + Mini ship-bundle
Browse files
README.md
CHANGED
|
@@ -107,7 +107,7 @@ PredictLM's TTT is an independent implementation of the published technique. Thi
|
|
| 107 |
|
| 108 |
- **Developed by**: ZeroOne Research
|
| 109 |
- **Distilled from**: [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) (v11.0)
|
| 110 |
-
- **Model card contact**:
|
| 111 |
- **License**: Apache 2.0 — permissive, commercial use allowed
|
| 112 |
|
| 113 |
## Why Mini (when to prefer this over Base)
|
|
@@ -201,8 +201,6 @@ Mini was trained via **warm-start sliced distillation**: a novel recipe for comp
|
|
| 201 |
|
| 202 |
The critical insight: distillation from scratch (Option A in our experiments) **failed to transfer to real OpenML data** — student matched teacher on synthetic but couldn't generalize. Warm-start sliced distillation (Option B, this release) succeeded because the student inherits the teacher's transfer ability as the starting point; distillation only needs to refine.
|
| 203 |
|
| 204 |
-
Reproducible from scratch with `scripts/train_v11_06_tiny.py` in the code repo.
|
| 205 |
-
|
| 206 |
## Intended use, limitations, ethical considerations
|
| 207 |
|
| 208 |
Identical to [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) — see that model card for full details:
|
|
@@ -222,30 +220,8 @@ The known weaknesses (cls below XGBoost; below TabPFN-2.5 / TabICLv2 on both axe
|
|
| 222 |
- **Training step**: 30,000 (final)
|
| 223 |
- **Training seed**: 42
|
| 224 |
- **Teacher**: `predictlm-base-26m` (v11.0)
|
| 225 |
-
- **Distillation recipe**: warm-start slice + online KL distillation
|
| 226 |
- **Eval-lock manifest SHA-256**: `fe4da8cccfc78fc3c7746579f604154af7d37e525c4fd575965ba77ce4fe0841` (identical to Base)
|
| 227 |
-
- **Code**: pinned at the v11.0 release tag of [zerooneresearch/predictlm-v11](https://github.com/zerooneresearch/predictlm-v11)
|
| 228 |
-
|
| 229 |
-
To reproduce from scratch:
|
| 230 |
-
|
| 231 |
-
```bash
|
| 232 |
-
# Pull the v11.0 teacher
|
| 233 |
-
huggingface-cli download zerooneresearch/predictlm-base-26m v11_final.pt --local-dir ./
|
| 234 |
-
|
| 235 |
-
# Reproduce Mini
|
| 236 |
-
python3 scripts/train_v11_06_tiny.py \
|
| 237 |
-
--teacher-ckpt v11_final.pt \
|
| 238 |
-
--warm-start-from-v11 v11_final.pt \
|
| 239 |
-
--mlp-variant gelu --norm-variant layernorm --share-factor 2 \
|
| 240 |
-
--corpus-manifest data/v11_05_corpus_manifest.json \
|
| 241 |
-
--copula-dir data/copulas \
|
| 242 |
-
--steps 30000 --batch-size 1 --accum-steps 8 \
|
| 243 |
-
--lr 3e-5 --min-lr 3e-6 --warmup-steps 500 \
|
| 244 |
-
--n-context-max 256 --n-query-max 64 \
|
| 245 |
-
--device cuda --precision fp16 \
|
| 246 |
-
--probe-interval 2500 --probe-warmup-steps 0 --probe-floor 0.30 \
|
| 247 |
-
--out-dir runs/v11_06_tiny_reproduce --seed 42
|
| 248 |
-
```
|
| 249 |
|
| 250 |
## Licensing
|
| 251 |
|
|
|
|
| 107 |
|
| 108 |
- **Developed by**: ZeroOne Research
|
| 109 |
- **Distilled from**: [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) (v11.0)
|
| 110 |
+
- **Model card contact**: message the org on the Hub
|
| 111 |
- **License**: Apache 2.0 — permissive, commercial use allowed
|
| 112 |
|
| 113 |
## Why Mini (when to prefer this over Base)
|
|
|
|
| 201 |
|
| 202 |
The critical insight: distillation from scratch (Option A in our experiments) **failed to transfer to real OpenML data** — student matched teacher on synthetic but couldn't generalize. Warm-start sliced distillation (Option B, this release) succeeded because the student inherits the teacher's transfer ability as the starting point; distillation only needs to refine.
|
| 203 |
|
|
|
|
|
|
|
| 204 |
## Intended use, limitations, ethical considerations
|
| 205 |
|
| 206 |
Identical to [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) — see that model card for full details:
|
|
|
|
| 220 |
- **Training step**: 30,000 (final)
|
| 221 |
- **Training seed**: 42
|
| 222 |
- **Teacher**: `predictlm-base-26m` (v11.0)
|
| 223 |
+
- **Distillation recipe**: warm-start slice + online KL distillation
|
| 224 |
- **Eval-lock manifest SHA-256**: `fe4da8cccfc78fc3c7746579f604154af7d37e525c4fd575965ba77ce4fe0841` (identical to Base)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 225 |
|
| 226 |
## Licensing
|
| 227 |
|