Instructions to use DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter

SGLang

How to use DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter with Docker Model Runner:
```
docker model run hf.co/DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter
```

'Make knowledge free for everyone'

Eval

The fine tuned model (DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit) has gained performace over the base model (unsloth/Llama-3.2-3B-Instruct-bnb-4bit) in the following tasks.

Test	Base Model	Fine-Tuned Model	Performance Gain
leaderboard_bbh_logical_deduction_seven_objects	0.2520	0.4360	0.1840
leaderboard_bbh_logical_deduction_five_objects	0.3560	0.4560	0.1000
leaderboard_musr_team_allocation	0.2200	0.3200	0.1000
leaderboard_bbh_disambiguation_qa	0.3040	0.3760	0.0720
leaderboard_gpqa_diamond	0.2222	0.2727	0.0505
leaderboard_bbh_movie_recommendation	0.5960	0.6360	0.0400
leaderboard_bbh_formal_fallacies	0.5080	0.5400	0.0320
leaderboard_bbh_tracking_shuffled_objects_three_objects	0.3160	0.3440	0.0280
leaderboard_bbh_causal_judgement	0.5455	0.5668	0.0214
leaderboard_bbh_web_of_lies	0.4960	0.5160	0.0200
leaderboard_math_geometry_hard	0.0455	0.0606	0.0152
leaderboard_math_num_theory_hard	0.0519	0.0649	0.0130
leaderboard_musr_murder_mysteries	0.5280	0.5400	0.0120
leaderboard_gpqa_extended	0.2711	0.2802	0.0092
leaderboard_bbh_sports_understanding	0.5960	0.6040	0.0080
leaderboard_math_intermediate_algebra_hard	0.0107	0.0143	0.0036

Framework versions

unsloth 2024.11.5
trl 0.12.0

Training HW

V100

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter

Base model

meta-llama/Llama-3.2-3B-Instruct

Quantized

unsloth/Llama-3.2-3B-Instruct-bnb-4bit

Finetuned

(342)

this model

Dataset used to train DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter

Collections including DevQuasar/analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit_adapter

LoRA Adapters

Collection

8 items • Updated Nov 29, 2024 • 1

Analytical Reasoning

Collection

Fine tuned models for Analytical Reasoning • 6 items • Updated Dec 1, 2024 • 2

Evaluation results

acc_norm on bbh
self-reported

0.417
acc_norm on gpqa
self-reported

0.269
exact_match on math
self-reported

0.087
acc_norm on mmlu
self-reported

0.282
acc_norm on musr
self-reported

0.365
acc on hellaswag
self-reported

0.514
acc_norm on hellaswag
self-reported

0.679