Instructions to use unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit")
model = AutoModelForCausalLM.from_pretrained("unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit

SGLang

How to use unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit with Docker Model Runner:
```
docker model run hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit
```

Error when load model

by lczazu - opened Jan 29, 2025

Discussion

lczazu

Jan 29, 2025

Hi, When I load this model by

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = config.checkpoint,load_in_4bit = load_in_4bit,
    max_seq_length = config.max_length,
    dtype = dtype,
)

Error happened

jimmy

Feb 13, 2025

Can you try this out?

model_name = "unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit"
max_seq_length = 2048
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=model_name,
max_seq_length=max_seq_length,
load_in_4bit=True,
)

For me, it works fine.

jimmy

Feb 13, 2025

Wow, you're on A800 80GB? So envy you! :) Hope it helps.

Matri1x

Feb 19, 2025

Wow, you're on A800 80GB? So envy you! :) Hope it helps.

Have you tried to load him with vllm?
I've encountered an error now.

assert param_data.shape == loaded_weight.shape
AssertionError

I don't know why it happened?

lczazu

Feb 19, 2025

Wow, you're on A800 80GB? So envy you! :) Hope it helps.

Now, I don’t have it anymore. 😅

jimmy

Feb 20, 2025

Wow, you're on A800 80GB? So envy you! :) Hope it helps.

Have you tried to load him with vllm?
I've encountered an error now.
assert param_data.shape == loaded_weight.shape
AssertionError
I don't know why it happened?
I believe I used vLLM per the following step I had.

As you can see, "use_vllm = True, # use vLLM for fast inference!" :)

Also the training did happened with no problem.
2 questions:

Can I see the whole errors which had "assert param_data.shape == loaded_weight.shape
AssertionError"?
Can you see the version of vLLM you're using?

jimmy

Feb 20, 2025

FYI
This is the summary of my training:

TrainOutput(global_step=250, training_loss=7.667776655330272e-05, metrics={'train_runtime': 3990.5973, 'train_samples_per_second': 0.063, 'train_steps_per_second': 0.063, 'total_flos': 0.0, 'train_loss': 7.667776655330272e-05})

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment