Instructions to use jburtoft/TencentARC-LLaMA-Pro-8B-Neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jburtoft/TencentARC-LLaMA-Pro-8B-Neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jburtoft/TencentARC-LLaMA-Pro-8B-Neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jburtoft/TencentARC-LLaMA-Pro-8B-Neuron")
model = AutoModelForCausalLM.from_pretrained("jburtoft/TencentARC-LLaMA-Pro-8B-Neuron")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jburtoft/TencentARC-LLaMA-Pro-8B-Neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jburtoft/TencentARC-LLaMA-Pro-8B-Neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jburtoft/TencentARC-LLaMA-Pro-8B-Neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/jburtoft/TencentARC-LLaMA-Pro-8B-Neuron

SGLang

How to use jburtoft/TencentARC-LLaMA-Pro-8B-Neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jburtoft/TencentARC-LLaMA-Pro-8B-Neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jburtoft/TencentARC-LLaMA-Pro-8B-Neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jburtoft/TencentARC-LLaMA-Pro-8B-Neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jburtoft/TencentARC-LLaMA-Pro-8B-Neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use jburtoft/TencentARC-LLaMA-Pro-8B-Neuron with Docker Model Runner:
```
docker model run hf.co/jburtoft/TencentARC-LLaMA-Pro-8B-Neuron
```

jburtoft commited on Jan 8, 2024

Commit

3f6f069

1 Parent(s): 15cc7c7

805754c77b7244f3088c4aebd7438f82c5f127c3ec1b0bf7a80333418dd0eb50

Browse files

Files changed (24) hide show

checkpoint/pytorch_model.bin/p114.model.layers.12.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p115.model.layers.12.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p116.model.layers.12.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p117.model.layers.12.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p118.model.layers.13.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p119.model.layers.13.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p12.model.layers.1.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p120.model.layers.13.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p121.model.layers.13.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p122.model.layers.13.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p123.model.layers.13.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p124.model.layers.13.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p125.model.layers.13.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p126.model.layers.13.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p127.model.layers.14.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p128.model.layers.14.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p129.model.layers.14.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p13.model.layers.1.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p130.model.layers.14.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p131.model.layers.14.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p132.model.layers.14.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p133.model.layers.14.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p134.model.layers.14.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p135.model.layers.14.post_attention_layernorm.weight +3 -0

checkpoint/pytorch_model.bin/p114.model.layers.12.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:df6dd630d100ab9b5baf2bcd5249d9d766598aea40f9061ae346dbe25c59f23c
+size 180355958

checkpoint/pytorch_model.bin/p115.model.layers.12.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1d3a12dcedc8a114e87cf6a6b22eeea2c08e7e254ec7fbe370766851ded72d5e
+size 180355964

checkpoint/pytorch_model.bin/p116.model.layers.12.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d6419b2d114ce03d299b782d770048f365300bb7f9ba365e3865ab59b7e23e2d
+size 17282

checkpoint/pytorch_model.bin/p117.model.layers.12.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c48405b7ee4df36eb2b426d0ab8fe9efc4cf8e2ef326ea4b5cda8e91ecef0138
+size 17309

checkpoint/pytorch_model.bin/p118.model.layers.13.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:32f75d2c8fe03bf9ae675bf8f05cfa21b20788c027741ec1d9bfcff180b3db61
+size 67109765

checkpoint/pytorch_model.bin/p119.model.layers.13.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cbc74f5365474463a9c620c6b259740d14948c7e0a365a4c6ec14d5a22456bed
+size 67109765

checkpoint/pytorch_model.bin/p12.model.layers.1.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c67935134cf42385cfc6ee23d9c41f3e0e146cf620fdfbfb7d61993332867779
+size 67109759

checkpoint/pytorch_model.bin/p120.model.layers.13.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b962dbcca96e461dae888041aa678054fa433bce132e239af9fe3647552f98b8
+size 67109765

checkpoint/pytorch_model.bin/p121.model.layers.13.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:72e6e3ca9945cfe94f6d7af517bef2cce3f2d5fee5d877e110508dd6f72337d2
+size 67109765

checkpoint/pytorch_model.bin/p122.model.layers.13.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:43f1b8f14b824efc8f7da54296990a527ef4fbe7923a661fc3939c357524f542
+size 180355964

checkpoint/pytorch_model.bin/p123.model.layers.13.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7c662c94be597e9d4e39107e78ee39bdae046cf76d35a41a8b647f4eb9d2a7ad
+size 180355958

checkpoint/pytorch_model.bin/p124.model.layers.13.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e3b7c15091338b92645cab0b62ec11112407a1c370b7db7fb81a176f1f02f57c
+size 180355964

checkpoint/pytorch_model.bin/p125.model.layers.13.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:750cf428d96d34edd179b9c52c07edc4e73532b6dbcf2f8b9eac6e6d66acc749
+size 17282

checkpoint/pytorch_model.bin/p126.model.layers.13.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:352163e6fa799c9fc9b6fc68a4d6bca9f99ded7aae4c0a4aa0f29ad8d2ccb339
+size 17309

checkpoint/pytorch_model.bin/p127.model.layers.14.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:63f8c809819c4a7f8cba02bd0e99c6285437202037a4f469513d8d64c528dedd
+size 67109765

checkpoint/pytorch_model.bin/p128.model.layers.14.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:81a98ae1c96afae978075208fedbe97e10e7ef82e11ce19a84acacc5a5641244
+size 67109765

checkpoint/pytorch_model.bin/p129.model.layers.14.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a9a6dd576e2e0ddeb873d80e0b350689982fe2a1ad3586b9fe848cfc9907c5e7
+size 67109765

checkpoint/pytorch_model.bin/p13.model.layers.1.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b774652267c782b295972d3ad5a80cd9b8726460798c63b155cbbed629d5ee95
+size 67109759

checkpoint/pytorch_model.bin/p130.model.layers.14.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f9af5bfaccaa489385826712e50a9d33420695ce7aaeceea5d7530fd03bfea7d
+size 67109765

checkpoint/pytorch_model.bin/p131.model.layers.14.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e62173af9865082f2c51d60b634a4e005cbef211474fbda81140a00bc681b1e6
+size 180355964

checkpoint/pytorch_model.bin/p132.model.layers.14.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:95db62f756486df4e1b1c34181bafa995a2c2f58cfe05d00f3354a97eef26831
+size 180355958

checkpoint/pytorch_model.bin/p133.model.layers.14.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:43835bd6efb1cec0ef54d06900b6b64e51af8e220fd9641c8ab45e5e74a83758
+size 180355964

checkpoint/pytorch_model.bin/p134.model.layers.14.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b70999a2af9a50edf6f5eb58a64e03a8d3598f5d9011b619b895ac3571ea9f48
+size 17282

checkpoint/pytorch_model.bin/p135.model.layers.14.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4e08439d520c6f70605a1848568f71b60f79d894bfaa8cd2638c66fafa550cd1
+size 17309