|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- Rowan/hellaswag |
|
|
- lighteval/piqa |
|
|
- google/boolq |
|
|
- omarmohamed/arc_easy |
|
|
metrics: |
|
|
- accuracy |
|
|
--- |
|
|
tags: |
|
|
- quantization |
|
|
- kronecker |
|
|
- second-order |
|
|
- YAQA |
|
|
- LLaMA |
|
|
- Qwen |
|
|
- efficient |
|
|
--- |
|
|
|
|
|
# ⚡ FastKronQuantization |
|
|
|
|
|
**Fast second-order Kronecker-factored quantization for LLMs** |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧠 Abstract |
|
|
|
|
|
Quantization with second-order information has shown strong promise for preserving model quality under aggressive compression. Building on the recent [YAQA](https://arxiv.org/abs/2505.22988) framework (Tseng et al., 2025b), which employs Kronecker-factored approximations of the Hessian via a power-iteration technique, we propose an alternative approach that replaces this step with a more efficient Kronecker decomposition method from Chekalina et al. (2025). |
|
|
This formulation preserves the benefits of second-order curvature-aware quantization while substantially reducing computational cost. |
|
|
We apply our method to **LLaMA-2 7B**, **LLaMA-3 8B Instruct**, and **Qwen-3 8B Instruct** and demonstrate that it achieves the same post-quantization model quality as YAQA, but with significantly faster computation — the Kronecker factors required for target quality are obtained with **10× fewer tokens** and approximately a **10× speedup** over the original work. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧭 Checkpoints |
|
|
|
|
|
| Model name | Architecture | Bits | |
|
|
|-------------------------|-------------------------|------| |
|
|
| `FastKronQuant-LLaMA2-7B-4bit` | LLaMA-2-7B | 4-bit | |
|
|
| `FastKronQuant-LLaMA3-8B-4bit` | LLaMA-3-8B-Instruct | 4-bit | |
|
|
| `FastKronQuant-Qwen3-8B-4bit` | Qwen-3-8B | 4-bit | |
|
|
| `FastKronQuant-LLaMA2-7B-2bit` | LLaMA-2-7B | 2-bit | |
|
|
| `FastKronQuant-LLaMA3-8B-2bit` | LLaMA-3-8B-Instruct | 2-bit | |
|
|
| `FastKronQuant-Qwen3-8B-2bit` | Qwen-3-8B | 2-bit | |
|
|
|
|
|
Each checkpoint is fully compatible with Hugging Face `transformers` and can be loaded like any standard model. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📌 Features |
|
|
|
|
|
- ⚡ **Fast Kronecker decomposition** — up to 10× faster factor estimation |
|
|
- 🧮 **Second-order quantization** — preserves model accuracy |
|
|
- 🪶 Works with popular architectures: **LLaMA-2**, **LLaMA-3**, **Qwen-3** |
|
|
- 🔸 Compatible with 🤗 `transformers` out of the box |
|
|
|
|
|
## 🚀 Usage Example (LLaMA-2 7B) |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
repo_id = "username/FastKronQuant-LLaMA2-7B" # replace with actual repo |
|
|
tokenizer = AutoTokenizer.from_pretrained(repo_id) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
repo_id, |
|
|
torch_dtype="auto", |
|
|
device_map="auto", |
|
|
) |
|
|
|
|
|
prompt = "What is the capital of France?" |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=50) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
## 🧪 Example — ARC Easy evaluation |
|
|
|
|
|
```python |
|
|
from datasets import load_dataset |
|
|
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
# Load ARC-Easy |
|
|
ds = load_dataset("ai2_arc", "ARC-Easy")["test"] |
|
|
|
|
|
# Load quantized model |
|
|
repo_id = "username/FastKronQuant-LLaMA2-7B" |
|
|
tok = AutoTokenizer.from_pretrained(repo_id) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
repo_id, |
|
|
torch_dtype="auto", |
|
|
device_map="auto" |
|
|
) |
|
|
pipe = pipeline("text-generation", model=model, tokenizer=tok) |
|
|
|
|
|
|
|
|
# Simple evaluation loop |
|
|
for i in range(3): |
|
|
q = ds[i]["question"] |
|
|
a = pipe(q, max_new_tokens=32)[0]["generated_text"] |
|
|
print(f"Q: {q}\nA: {a}\n---") |
|
|
``` |
|
|
|
|
|
If you use model chekpoints in your experiments, please cite |
|
|
>@misc{chekalina2025gfwsvd, |
|
|
> title={Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models}, |
|
|
> author={Viktoriia Chekalina and Daniil Moskovskiy and Daria Cherniuk and Maxim Kurkin and Andrey Kuznetsov and Evgeny Frolov}, |
|
|
> year={2025}, |
|
|
> eprint={2505.17974}, |
|
|
> archivePrefix={arXiv}, |
|
|
> primaryClass={cs.LG}, |
|
|
> url={https://arxiv.org/abs/2505.17974}, |
|
|
>} |
|
|
|
|
|
## 📊 Zero-shot results — LLaMA-3 8B |
|
|
|
|
|
### 🟡 4-bit Quantization |
|
|
|
|
|
| Method | Steps | ARC_c ↑ | BoolQ ↑ | PIQA ↑ | ARC_e ↑ | HSwag ↑ | AVG ↑ | GPU/h ↓ | Tokens ↓ | |
|
|
|---------------------|-------|---------|---------|--------|---------|---------|--------|---------|-----------| |
|
|
| 16 bit (baseline) | – | **0.5171** | **0.8409** | **0.7986** | **0.8177** | **0.5908** | **0.7131** | – | – | |
|
|
| 4-bit Sketch A | 4096 | **0.5136** | **0.8443** | 0.7997 | 0.8198 | **0.5865** | 0.7127 | 92 | 16 M | |
|
|
| 4-bit FastKron | 75 | 0.5116 | 0.8438 | **0.8025** | **0.8207** | 0.5863 | **0.7129** | 9.5 | 712 K | |
|
|
| 4-bit No Hess | – | 0.5119 | 0.8415 | 0.7959 | 0.8097 | 0.5859 | 0.7112 | – | – | |
|
|
|
|
|
|
|
|
### 🟠 2-bit Quantization |
|
|
|
|
|
| Method | Steps | ARC_c ↑ | BoolQ ↑ | PIQA ↑ | ARC_e ↑ | HSwag ↑ | AVG ↑ | GPU/h ↓ | Tokens ↓ | |
|
|
|---------------------|-------|---------|---------|--------|---------|---------|--------|---------|-----------| |
|
|
| 2-bit Sketch A | 4096 | **0.4312** | 0.7567 | 0.7647 | 0.7391 | **0.5259** | 0.6435 | 92 | 16 M | |
|
|
| 2-bit FastKron | 100 | 0.4277 | **0.7646** | **0.7661** | **0.7468** | 0.5159 | **0.6442** | 11.5 | 950 K | |
|
|
| 2-bit No Hess | – | 0.2363 | 0.6336 | 0.6554 | 0.5108 | 0.3620 | 0.5094 | – | – | |
|
|
|
|
|
|
|
|
|
|
|
## 📊 Zero-shot results — Qwen-3 8B |
|
|
|
|
|
### 🟡 4-bit Quantization |
|
|
|
|
|
| Method | Steps | ARC_c ↑ | BoolQ ↑ | PIQA ↑ | ARC_e ↑ | HSwag ↑ | AVG ↑ | GPU/h ↓ | Tokens ↓ | |
|
|
|---------------------|-------|---------|---------|--------|---------|---------|--------|---------|-----------| |
|
|
| 16 bit (baseline) | – | **0.5563** | **0.8682** | **0.7677** | **0.8354** | **0.5708** | **0.7197** | – | – | |
|
|
| 4-bit Sketch A | 4096 | **0.5503** | 0.8611 | 0.7612 | 0.8324 | 0.5601 | **0.7132** | 84 | 8 M | |
|
|
| 4-bit FastKron | 150 | 0.5469 | 0.8667 | 0.7601 | **0.8287** | **0.5637** | **0.7132** | 42 | 712 K | |
|
|
| 4-bit No Hess | – | 0.5467 | **0.8675** | **0.7622** | 0.8312 | 0.5585 | **0.7132** | – | – | |
|
|
|
|
|
|
|
|
### 🟠 2-bit Quantization |
|
|
|
|
|
| Method | Steps | ARC_c ↑ | BoolQ ↑ | PIQA ↑ | ARC_e ↑ | HSwag ↑ | AVG ↑ | GPU/h ↓ | Tokens ↓ | |
|
|
|---------------------|-------|---------|---------|--------|---------|---------|--------|---------|-----------| |
|
|
| 2-bit Sketch A | 4096 | 0.4536 | 0.7782 | **0.7435** | **0.7797** | 0.4611 | 0.6432 | 84 | 8 M | |
|
|
| 2-bit FastKron | 150 | **0.4616** | 0.8416 | 0.7334 | 0.7702 | **0.4853** | **0.6584** | 42 | 712 K | |
|
|
| 2-bit No Hess | – | 0.3993 | **0.8675** | 0.7743 | 0.7003 | 0.4758 | 0.6434 | – | – | |
|
|
|
|
|
|
|
|
## 📊 Zero-shot results — LLaMA-2 7B |
|
|
|
|
|
### 🟡 4-bit Quantization |
|
|
|
|
|
| Method | Steps | ARC_c ↑ | BoolQ ↑ | PIQA ↑ | ARC_e ↑ | HSwag ↑ | AVG ↑ | GPU/h ↓ | Tokens ↓ | |
|
|
|---------------------|-------|---------|---------|--------|---------|---------|--------|---------|-----------| |
|
|
| 16 bit (baseline) | – | **0.4325** | **0.7767** | **0.7774** | **0.7617** | **0.5721** | **0.6640** | – | – | |
|
|
| 4-bit Sketch A | 4096 | 0.4274 | 0.7688 | 0.7752 | **0.7613** | **0.5672** | 0.6599 | 50 | 16 M | |
|
|
| 4-bit FastKron | 75 | 0.4283 | 0.7792 | **0.7802** | 0.7610 | 0.5660 | 0.6629 | 5 | 712 K | |
|
|
| 4-bit No Hess | – | **0.4352** | **0.7875** | 0.7742 | 0.7609 | 0.5628 | **0.6641** | – | – | |
|
|
|
|
|
|
|
|
### 🟠 2-bit Quantization |
|
|
|
|
|
| Method | Steps | ARC_c ↑ | BoolQ ↑ | PIQA ↑ | ARC_e ↑ | HSwag ↑ | AVG ↑ | GPU/h ↓ | Tokens ↓ | |
|
|
|---------------------|-------|---------|---------|--------|---------|---------|--------|---------|-----------| |
|
|
| 2-bit Sketch A | 4096 | 0.3805 | 0.7333 | 0.7562 | **0.7192** | **0.5227** | 0.6223 | 50 | 16 M | |
|
|
| 2-bit FastKron | 150 | **0.3843** | **0.7510** | **0.7600** | 0.7112 | 0.5139 | **0.6240** | 6 | 1400 K | |
|
|
| 2-bit No Hess | – | 0.2210 | 0.6355 | 0.6306 | 0.5152 | 0.3422 | 0.4689 | – | – | |
|
|
|
