IFM/LoopedLM-P1-dense-d112-336b
Text Generation • 4B • Updated • 130
Towards Looped Models Done Right
Looped language model checkpoints from Part I and Part II (Rethinking at Fixed Points). Native xLLM format; loading requires the xLLM code.
Note dense_d112 · 336B tokens
IFM/LoopedLM-P1-dense-d112-500b
Text Generation • 4B • Updated • 125Note dense_d112 · 500B tokens
IFM/LoopedLM-P1-dense-d28-336b
Text Generation • 1B • Updated • 141Note dense_d28 · 336B tokens
IFM/LoopedLM-P1-dense-huginn-336b
Text Generation • 1B • Updated • 135Note dense_huginn · 336B tokens
IFM/LoopedLM-P1-dense-huginn-500b
Text Generation • 1B • Updated • 124Note dense_huginn · 500B tokens
IFM/LoopedLM-P1-dense-middle-336b
Text Generation • 1B • Updated • 136Note dense_middle · 336B tokens
IFM/LoopedLM-P1-dense-middle-prelude-injection-336b
Text Generation • 1B • Updated • 128Note dense_middle_prelude_injection · 336B tokens
IFM/LoopedLM-P1-dense-ouro-336b
Text Generation • 1B • Updated • 142Note dense_ouro · 336B tokens
IFM/LoopedLM-P1-dense-ouro-raw-injection-336b
Text Generation • 1B • Updated • 132Note dense_ouro_raw_injection · 336B tokens
IFM/LoopedLM-P1-dense-shared-hl-336b
Text Generation • 1B • Updated • 132Note dense_shared_hl · 336B tokens
IFM/LoopedLM-P1-dense-split-hl-336b
Text Generation • 1B • Updated • 132Note dense_split_hl · 336B tokens
IFM/LoopedLM-P1-moe-ff112-336b
Text Generation • 33B • Updated • 139Note moe_ff112 · 336B tokens
IFM/LoopedLM-P1-moe-ff112-500b
Text Generation • 33B • Updated • 136Note moe_ff112 · 500B tokens
IFM/LoopedLM-P1-moe-ff112-prelude-injection-336b
Text Generation • 33B • Updated • 142Note moe_ff112_prelude_injection · 336B tokens
IFM/LoopedLM-P1-moe-huginn-336b
Text Generation • 9B • Updated • 142Note moe_huginn · 336B tokens
IFM/LoopedLM-P1-moe-huginn-500b
Text Generation • 9B • Updated • 142Note moe_huginn · 500B tokens
IFM/LoopedLM-P1-moe-middle-336b
Text Generation • 9B • Updated • 146Note moe_middle · 336B tokens
IFM/LoopedLM-P1-moe-middle-500b
Text Generation • 9B • Updated • 142Note moe_middle · 500B tokens
IFM/LoopedLM-P1-moe-middle-input-injection-336b
Text Generation • 9B • Updated • 142Note moe_middle_input_injection · 336B tokens
IFM/LoopedLM-P1-moe-middle-input-injection-500b
Text Generation • 9B • Updated • 147Note moe_middle_input_injection · 500B tokens
IFM/LoopedLM-P1-moe-ouro-336b
Text Generation • 9B • Updated • 155Note moe_ouro · 336B tokens
IFM/LoopedLM-P1-moe-ouro-500b
Text Generation • 9B • Updated • 147Note moe_ouro · 500B tokens
IFM/LoopedLM-P1-moe-shared-hl-336b
Text Generation • 9B • Updated • 150Note moe_shared_hl · 336B tokens
IFM/LoopedLM-P1-moe-shared-hl-500b
Text Generation • 9B • Updated • 151Note moe_shared_hl · 500B tokens
IFM/LoopedLM-P1-dense-split-hl-x2-336b
Text Generation • 1B • Updated • 137Note dense_split_hl_x2 · 336B tokens
IFM/LoopedLM-P2-d4-m
Text Generation • 0.8B • Updated • 282Note Table 4, M, D4
IFM/LoopedLM-P2-d4-s
Text Generation • 0.3B • Updated • 298Note Table 4, S, D4
IFM/LoopedLM-P2-huginn-l-fixed-pln5
Text Generation • 2B • Updated • 297Note Table 2, L, Fixed PLN-5
IFM/LoopedLM-P2-huginn-l-fixed-r5
Text Generation • 2B • Updated • 298Note Table 2, L, Fixed R=5
IFM/LoopedLM-P2-huginn-l-learned-entropy0
Text Generation • 2B • Updated • 305Note Table 2, L, Learned lambda_H=0
IFM/LoopedLM-P2-huginn-l-learned-entropy0p01
Text Generation • 2B • Updated • 320Note Table 2, L, Learned lambda_H=0.01
IFM/LoopedLM-P2-huginn-m-fixed-pln5
Text Generation • 0.8B • Updated • 318Note Table 2, M, Fixed PLN-5
IFM/LoopedLM-P2-huginn-m-fixed-r5
Text Generation • 0.8B • Updated • 332Note Table 2, M, Fixed R=5
IFM/LoopedLM-P2-huginn-m-huginn-linear
Text Generation • 0.8B • Updated • 323Note Table 3, M, Huginn-Linear
IFM/LoopedLM-P2-huginn-m-learned-entropy0
Text Generation • 0.8B • Updated • 323Note Table 2, M, Learned lambda_H=0
IFM/LoopedLM-P2-huginn-m-learned-entropy0p01
Text Generation • 0.8B • Updated • 342Note Table 2, M, Learned lambda_H=0.01 (Table 4 M teacher)
IFM/LoopedLM-P2-huginn-m-orthoinj
Text Generation • 0.8B • Updated • 339Note Table 3, M, OrthoInj
IFM/LoopedLM-P2-huginn-m-parcae
Text Generation • 0.8B • Updated • 335Note Table 3, M, Parcae-Decay
IFM/LoopedLM-P2-huginn-s-fixed-pln5
Text Generation • 0.3B • Updated • 345Note Table 2, S, Fixed PLN-5
IFM/LoopedLM-P2-huginn-s-fixed-r5
Text Generation • 0.3B • Updated • 345Note Table 2, S, Fixed R=5
IFM/LoopedLM-P2-huginn-s-huginn-linear
Text Generation • 0.3B • Updated • 344Note Table 3, S, Huginn-Linear
IFM/LoopedLM-P2-huginn-s-learned-entropy0
Text Generation • 0.3B • Updated • 344Note Table 2, S, Learned lambda_H=0
IFM/LoopedLM-P2-huginn-s-learned-entropy0p01
Text Generation • 0.3B • Updated • 346Note Table 2, S, Learned lambda_H=0.01 (Table 4 S teacher)
IFM/LoopedLM-P2-huginn-s-orthoinj
Text Generation • 0.3B • Updated • 352Note Table 3, S, OrthoInj
IFM/LoopedLM-P2-huginn-s-parcae
Text Generation • 0.3B • Updated • 352Note Table 3, S, Parcae-Decay
IFM/LoopedLM-P2-distilled-m-random-init
0.2B • Updated • 53Note Table 4, M, Distilled, random init
IFM/LoopedLM-P2-distilled-m-teacher-init
0.2B • Updated • 54Note Table 4, M, Distilled, teacher init
IFM/LoopedLM-P2-distilled-s-random-init
54.3M • Updated • 44Note Table 4, S, Distilled, random init
IFM/LoopedLM-P2-distilled-s-teacher-init
54.3M • Updated • 56Note Table 4, S, Distilled, teacher init