01RAI commited on
Commit
b846449
·
verified ·
1 Parent(s): 950834d

PredictLM v11.0 + Mini ship-bundle

Browse files
Files changed (1) hide show
  1. README.md +2 -26
README.md CHANGED
@@ -107,7 +107,7 @@ PredictLM's TTT is an independent implementation of the published technique. Thi
107
 
108
  - **Developed by**: ZeroOne Research
109
  - **Distilled from**: [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) (v11.0)
110
- - **Model card contact**: open an issue at the [code repo](https://github.com/zerooneresearch/predictlm-v11) or message the org on the Hub
111
  - **License**: Apache 2.0 — permissive, commercial use allowed
112
 
113
  ## Why Mini (when to prefer this over Base)
@@ -201,8 +201,6 @@ Mini was trained via **warm-start sliced distillation**: a novel recipe for comp
201
 
202
  The critical insight: distillation from scratch (Option A in our experiments) **failed to transfer to real OpenML data** — student matched teacher on synthetic but couldn't generalize. Warm-start sliced distillation (Option B, this release) succeeded because the student inherits the teacher's transfer ability as the starting point; distillation only needs to refine.
203
 
204
- Reproducible from scratch with `scripts/train_v11_06_tiny.py` in the code repo.
205
-
206
  ## Intended use, limitations, ethical considerations
207
 
208
  Identical to [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) — see that model card for full details:
@@ -222,30 +220,8 @@ The known weaknesses (cls below XGBoost; below TabPFN-2.5 / TabICLv2 on both axe
222
  - **Training step**: 30,000 (final)
223
  - **Training seed**: 42
224
  - **Teacher**: `predictlm-base-26m` (v11.0)
225
- - **Distillation recipe**: warm-start slice + online KL distillation, see `scripts/train_v11_06_tiny.py`
226
  - **Eval-lock manifest SHA-256**: `fe4da8cccfc78fc3c7746579f604154af7d37e525c4fd575965ba77ce4fe0841` (identical to Base)
227
- - **Code**: pinned at the v11.0 release tag of [zerooneresearch/predictlm-v11](https://github.com/zerooneresearch/predictlm-v11)
228
-
229
- To reproduce from scratch:
230
-
231
- ```bash
232
- # Pull the v11.0 teacher
233
- huggingface-cli download zerooneresearch/predictlm-base-26m v11_final.pt --local-dir ./
234
-
235
- # Reproduce Mini
236
- python3 scripts/train_v11_06_tiny.py \
237
- --teacher-ckpt v11_final.pt \
238
- --warm-start-from-v11 v11_final.pt \
239
- --mlp-variant gelu --norm-variant layernorm --share-factor 2 \
240
- --corpus-manifest data/v11_05_corpus_manifest.json \
241
- --copula-dir data/copulas \
242
- --steps 30000 --batch-size 1 --accum-steps 8 \
243
- --lr 3e-5 --min-lr 3e-6 --warmup-steps 500 \
244
- --n-context-max 256 --n-query-max 64 \
245
- --device cuda --precision fp16 \
246
- --probe-interval 2500 --probe-warmup-steps 0 --probe-floor 0.30 \
247
- --out-dir runs/v11_06_tiny_reproduce --seed 42
248
- ```
249
 
250
  ## Licensing
251
 
 
107
 
108
  - **Developed by**: ZeroOne Research
109
  - **Distilled from**: [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) (v11.0)
110
+ - **Model card contact**: message the org on the Hub
111
  - **License**: Apache 2.0 — permissive, commercial use allowed
112
 
113
  ## Why Mini (when to prefer this over Base)
 
201
 
202
  The critical insight: distillation from scratch (Option A in our experiments) **failed to transfer to real OpenML data** — student matched teacher on synthetic but couldn't generalize. Warm-start sliced distillation (Option B, this release) succeeded because the student inherits the teacher's transfer ability as the starting point; distillation only needs to refine.
203
 
 
 
204
  ## Intended use, limitations, ethical considerations
205
 
206
  Identical to [predictlm-base-26m](https://huggingface.co/zerooneresearch/predictlm-base-26m) — see that model card for full details:
 
220
  - **Training step**: 30,000 (final)
221
  - **Training seed**: 42
222
  - **Teacher**: `predictlm-base-26m` (v11.0)
223
+ - **Distillation recipe**: warm-start slice + online KL distillation
224
  - **Eval-lock manifest SHA-256**: `fe4da8cccfc78fc3c7746579f604154af7d37e525c4fd575965ba77ce4fe0841` (identical to Base)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
225
 
226
  ## Licensing
227