Instructions to use UsernameJustAnother/Nemo-12B-Marlin-v5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use UsernameJustAnother/Nemo-12B-Marlin-v5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="UsernameJustAnother/Nemo-12B-Marlin-v5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("UsernameJustAnother/Nemo-12B-Marlin-v5")
model = AutoModelForCausalLM.from_pretrained("UsernameJustAnother/Nemo-12B-Marlin-v5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use UsernameJustAnother/Nemo-12B-Marlin-v5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "UsernameJustAnother/Nemo-12B-Marlin-v5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "UsernameJustAnother/Nemo-12B-Marlin-v5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/UsernameJustAnother/Nemo-12B-Marlin-v5

SGLang

How to use UsernameJustAnother/Nemo-12B-Marlin-v5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "UsernameJustAnother/Nemo-12B-Marlin-v5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "UsernameJustAnother/Nemo-12B-Marlin-v5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "UsernameJustAnother/Nemo-12B-Marlin-v5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "UsernameJustAnother/Nemo-12B-Marlin-v5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use UsernameJustAnother/Nemo-12B-Marlin-v5 with Docker Model Runner:
```
docker model run hf.co/UsernameJustAnother/Nemo-12B-Marlin-v5
```

Congrats!

by aaronday3 - opened Aug 15, 2024

Discussion

aaronday3

Aug 15, 2024

It's great to see someone else being inspired by my work, you might want to reduce the learning rate a bit, maybe 4e-6, depending on what your eval loss looks like.

I would heavily suggest turning on eval loss, even using something like 1% of your dataset, as without it, you are going in blind.

Feel free to ask me for help, I'm on my discord server and Kobold. Also curious what exactly is in your dataset, and which human conversations you are referring to :P

UsernameJustAnother

Owner Aug 15, 2024

Yeah, I'm probably going to show up on the Discord once I feel like I'm not a complete idiot - but seriously, it was the write-up for Celeste that made me think I could do this. Huge thanks for that!

Since I trained v5 I've managed to work out eval loss and get stats going to wandb. You're not wrong, it makes a big difference, and now I'm able to do some real comparison testing of learning rates and rank/alpha.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment