rostlabs/rost-1b-instruct

rost is a bilingual Romanian/English language model trained from scratch, with a purpose-built Romanian tokenizer rather than one inherited from an English model. This repository holds the instruction-tuned chat model.

Model overview

parameters	1.384B total (24 layers, 1,536 hidden, 12 heads)
context length	4,096 tokens
vocabulary	32,768, bilingual -- rostlabs/rost-tok-bilingual
position encoding	RoPE, theta 100,000
attention	sliding-window pattern SSSL; 18 of 24 layers see a quarter context
precision on disk	float32 safetensors. Load as bfloat16 -- that is what it trained in
stage	sft, checkpoint step 000387
languages	Romanian (primary), English

Not a Llama or Mistral derivative: the architecture is its own, so the modelling code ships in this repository and trust_remote_code=True is required.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "rostlabs/rost-1b-instruct", trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()
tokenizer = AutoTokenizer.from_pretrained("rostlabs/rost-1b-instruct")

messages = [{"role": "user", "content": "Care este capitala Romaniei?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(inputs["input_ids"].to("cuda"), max_new_tokens=128)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

dtype=torch.bfloat16 is worth passing explicitly. The weights are stored as float32 because the release verification compares logits exactly and that comparison is only meaningful in float32; the model was trained in bfloat16 and loses nothing by being loaded that way.

Measured on a single RTX 5070 (12 GB, SDPA path): ~91 tokens/second, ~3.3 GB resident.

Recommended sampling parameters

generation_config.json carries these, so model.generate() uses them without being asked:

{
  "do_sample": true,
  "temperature": 0.6,
  "top_k": 50,
  "repetition_penalty": 1.1,
  "eos_token_id": [32763, 32759]
}

The repetition penalty is not decoration. Measured over 198 generations of 400 tokens, on prompts chosen to provoke the failure:

decoding	replies that looped	worst repeated 6-gram
greedy	50%	44x
temperature 0.2, no penalty	44%	44x
temperature 0.6, no penalty	22%	10x
temperature 0.6, penalty 1.1	0%	2x

A penalty of 1.1 also raises the share of replies that end by emitting their stop token, rather than running out of budget, from 50% to 83%. Stronger settings suppress repetition further and cost accuracy: at 1.3 the model stopped looping and began inventing etymologies, so 1.1 is the mildest setting that works and that is why it is the default.

Before this revision the repository set no eos_token_id in any file, so generate() had no stop condition and ran to max_new_tokens on every call.

Not available

GGUF, and therefore Ollama, LM Studio and llama.cpp. llama.cpp compiles architectures in rather than loading them dynamically, so support requires this architecture implemented and upstreamed there. It is planned, not done.

Tool and function calling. The instruct checkpoint has had conversational fine-tuning only. No agent fine-tuning has been applied, so the tool tokens are not reachable under any prompt framing. Do not build an agent on this checkpoint yet.

Training

tokens	11.68B, single pass -- no data was repeated
optimizer steps	11,136 at a 1,048,576-token batch
hardware	8x H100 80GB, ~3.6 hours
schedule	warmup-stable-decay, decay over the final 30%
precision	bfloat16 with FP8 matmuls

Trained in two phases, which is the substance of the recipe rather than a detail:

	Romanian	English (ClimbMix)	DQA	code
phase 1, steps 0-7,795	30%	60%	5%	5%
phase 2, steps 7,795-11,136	55%	35%	10%	--

The second phase raises Romanian while the learning rate decays, so the model finishes on a Romanian-heavy diet. A d6 rehearsal measured Romanian improving roughly 8x faster than English across that phase.

Data

source	licence
Romanian: FineWeb2-ro, educational score >= 3, diacritic-normalised	ODC-BY
English: Nemotron ClimbMix	CC-BY-NC-4.0
High-quality QA: Nemotron-CC-v2.1 HQ-DQA	gated
Code: Nemotron-CC-Code-v1 (phase 1 only)	gated
Chat fine-tuning: OpenLLM-Ro sets	CC-BY-NC-4.0

Evaluation

OpenLLM-Ro suite, base checkpoint, 400 rows per task, zero-shot, scored by likelihood over the options:

task	accuracy	normalised	chance
ro_hellaswag	31.25	38.50	25.0
ro_truthfulqa (MC1)	21.25	34.00	14.3
ro_arc_challenge	24.50	30.25	28.6
ro_mmlu	28.50	29.75	25.0
ro_winogrande	52.25	52.25	50.0
mean (normalised)		36.95	28.6

Read these against the chance column, not on their own. The model is clearly above chance on sentence completion and truthfulness, and close to chance on ro_arc_challenge and ro_winogrande. At 400 rows a task, differences under about 5 points are inside the noise.

These are not comparable to the OpenLLM-Ro leaderboard. Published figures there average each task over several few-shot settings; these are zero-shot, which understates them. For scale rather than ranking: RoLlama2-7b-Base reports a 42.05 four-task accuracy average against this model's 34.12 -- from a model 5x larger trained on far more data.

Also measured: CORE metric 0.2450 on the base model (0.1608 at step 2,000), and 0.3021 validation bits-per-byte for the instruct checkpoint against 0.5113 for the base model it started from.

Intended use

Research on Romanian language modelling, Romanian text generation and completion, and as a base for further fine-tuning. It is small enough to run on a consumer GPU, which is the point.

Out of scope: anything requiring factual reliability, agent or tool use, long-context work beyond 4,096 tokens, commercial deployment (see the licence), and any decision affecting a person's rights, health, safety or finances.

Limitations

• It confabulates confidently. Asked about Bucharest it correctly names the capital and then places it in the wrong county. Specifics need checking.
• It repeats. Restating a sentence with the clauses swapped is a common failure, inherited from the base model and only partly removed by fine-tuning. Decode with the shipped repetition_penalty of 1.1; with penalties off and a low temperature it will repeat one sentence until it runs out of tokens.
• Reasoning is near chance. See ro_arc_challenge and ro_winogrande above.
• 4,096 tokens of context, well short of contemporary models.
• Domain skew. The Romanian pretraining data is roughly one third health content by character count, with history, geography, finance and education each near 10%, and entertainment, gaming and software each under 1%. Fluency is uneven accordingly.
• The Romanian corpus is internally duplicated, which inflates absolute Romanian bits-per-byte by around 0.09. Comparisons on fixed sets are unaffected.
• No safety tuning of any kind has been applied.

Licence

CC-BY-NC-4.0, non-commercial. Both halves of the training data carry non-commercial terms -- ClimbMix is CC-BY-NC-4.0 ("for research and development only") and the OpenLLM-Ro fine-tuning sets are CC-BY-NC-4.0 -- and the model inherits them. The ClimbMix mirror used is tagged MIT, which does not override the upstream terms. The Romanian data is ODC-BY and requires attribution to FineWeb2.

Citation

@misc{rost2026,
  title  = {rost: a bilingual Romanian-English language model trained from scratch},
  author = {Iancu, Stefan},
  year   = {2026},
  url    = {https://huggingface.co/rostlabs/rost-1b-instruct}
}

The original training checkpoint (model_*.pt, meta_*.json) ships alongside the safetensors, for use with the rost repository.