Instructions to use USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S
# Run inference directly in the terminal:
llama cli -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S
# Run inference directly in the terminal:
llama cli -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S
# Run inference directly in the terminal:
./llama-cli -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S

Use Docker

docker model run hf.co/USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S

LM Studio
Jan

vLLM

How to use USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S

Ollama
How to use USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf with Ollama:
```
ollama run hf.co/USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S
```
Unsloth Desktop
Docker Model Runner
How to use USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf with Docker Model Runner:
```
docker model run hf.co/USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S
```

Lemonade

How to use USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull USTC-KnowledgeComputingLab/Llama3-KALE-LM-Chem-8B-gguf:Q4_K_S

Run and chat with the model

lemonade run user.Llama3-KALE-LM-Chem-8B-gguf-Q4_K_S

List all available models

lemonade list

Atomic Chat

2985mN1 commited on Nov 7, 2024

Commit

aef3f84

verified ·

1 Parent(s): 1d17f0d

sync e81c954

Browse files

Files changed (11) hide show

Llama3-KALE-LM-Chem-8B-F16.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q2_K.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q3_K_S.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q4_0.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q4_1.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q4_K_S.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q5_0.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q5_1.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q5_K_S.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q6_K.gguf +1 -1
Llama3-KALE-LM-Chem-8B-Q8_0.gguf +1 -1

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95e47b24c77483677689f13d0480c3a01befa504e08f085a065e9db1f43457ab
 size 16068891424

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7e1de8d8bf50ad24d633a7622c7947013f1ada7e58ba5ee4de28d2add8032753
 size 3179131680

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a046218c314bc0ce379953fe3d60808af47327d36d8851220222589d5126f8fa
 size 3664499488

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:35730462a2bc465e074736731e872f651f564932a61b2e619ec4c63cf8a779cc
 size 4661211936

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d4f1907945bf447404257dd803f05878e95266c35ac782a26b0af50e2e58b34b
 size 5130253088

 version https://git-lfs.github.com/spec/v1
+oid sha256:ce347a849e91f558ce36fddd38765df06b700d50585b488b15e1baa05fe1251f
 size 4692669216

 version https://git-lfs.github.com/spec/v1
+oid sha256:95f44ce7439d98701da1c60ad8f54fdc9dbcc5a0d65acdc0660385f4572b1e56
 size 5599294240

 version https://git-lfs.github.com/spec/v1
+oid sha256:d11ec029e326e5fcc14b2c03b1819d24c270624a7e49dbca987620cd6007fc4f
 size 6068335392

 version https://git-lfs.github.com/spec/v1
+oid sha256:95c1c4fa983bc257cc76c0bf7ec3a76e4d6463bca0d901eba0fdb10e3cd18b3b
 size 6596006688

 version https://git-lfs.github.com/spec/v1
+oid sha256:281532b3ca4c39dfa640714d0fcbfbe09191fb54d694c624c411966cff5c7551
 size 8540771104