steampunque/Qwen3.5-35B-A3B-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3.5-35B-A3B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.5-35B-A3B

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 8/26/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOSD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_6666"],[1 ,"Q5_K_6655"],[2 ,"Q4_K_6655"],[3 ,"Q4_K_6655"],
   [4 ,"Q4_K_5555"],[5 ,"Q4_K_5555"],[6 ,"Q4_K_5555"],[7 ,"Q4_K_6555"],
   [8 ,"Q4_K_5554"],[9 ,"Q4_K_5554"],[10,"Q4_K_5554"],[11,"Q4_K_6554"],
   [12,"Q4_K_5554"],[13,"Q4_K_5554"],[14,"Q4_K_5554"],[15,"Q4_K_6554"],
   [16,"Q4_K_6554"],[17,"Q4_K_6554"],[18,"Q4_K_6554"],[19,"Q4_K_6554"],
   [20,"Q4_K_6554"],[21,"Q4_K_6554"],[22,"Q4_K_6554"],[23,"Q4_K_6665"],
   [24,"Q4_K_6554"],[25,"Q4_K_6554"],[26,"Q4_K_6554"],[27,"Q4_K_6665"],
   [28,"Q4_K_6554"],[29,"Q4_K_6554"],[30,"Q4_K_6554"],[31,"Q4_K_6665"],
   [32,"Q4_K_6665"],[33,"Q4_K_6665"],[34,"Q4_K_6665"],[35,"Q4_K_6665"],
   [36,"Q5_K_6665"],[37,"Q5_K_6666"],[38,"Q5_K_6668"],[39,"Q6_K_8666"],
   [40,"Q4_K_6554"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The quant was sized to allow running it fully offloaded in 24G VRAM with some room left for both vision tower and context. The layer quants were optimized for essentially 100% success rate across a curated set of reasoning test prompts using greedy deterministic sampling and cuda backend for very fast token gen. The minimum quant across layers is Q4_K to minimize quantization entropy loss. The quant includes a ~0.4B Q4_K MTP layer.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	21.7e9	6.8	Hybrid quant with Q6_K embedding Q6_K output with ~0.4B MTP layer
Q4_E_H	21.8e9	6.7	Extended MP quant with Q6_K embedding/output and ~0.4B MTP layer

Usage:

Qwen3.5-35B-A3B is a vision capable moe RL model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Due to the attention scheme used in the model, straightforward speculation approaches cannot be used. Later versions of llama.cpp as of 4/23/2026 attempt to circumvent this problem however efficiency will not be good so use of speculation prior to Qwen3.5 MTP patches is not recommended. As of llama.cpp b9180 MTP support for the model was added to upstream and may be experimented with by using the Q4_E_H_MTP quant.

The model can be run fully offloaded into 24G VRAM, or with CPU and expert layer offload via config OT="-ot exps=CPU -ngl 99" Because the model is a 3B active moe the CPU expert offload still gives good gen rate with very large context available.

On a 9900k/4070 or 2x 4070 setup (1 RPC) approx performance (llama.cpp b10444) with no MTP is:

CONFIG	QKV	NKV	gen tps
4070+ 9900k CPU exp offload	F16	470k+	31
4070+ 9900k CPU exp offload	Q8_0	760k+	31
2x4070 (RPC)	F16	32k +	85
2x4070 (RPC) + vision tower	F16	32k +	85
2x4070 (RPC)	Q8_0	64k +	64

High context yarn config is as follows: set base context for yarn rope scale compute to 262144 (256k), then with a context of 470k tokens the rope scale = 470 / 256 = 1.83.

Then on model start pass --rope-scaling yarn --yarn-orig-ctx 262144 --rope_scale 1.83 (must be ajusted if kv other than 470k)

Later versions of llama.cpp have a bug which soft caps context length to the training context, effectively disabling yarn context extension. Patch server-context.cpp according to https://github.com/ggml-org/llama.cpp/issues/22140 to fix it.

The model appears to be trained to decide itself whether to do a think block or not. When it does a think block it sometimes falls into heavy overthinking but does come up with accurate answers. Over a curated set of eval prompts the model did exceptionally well. To avoid the overthinking inject think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into a think block inject a bootstrap think start following the assistant prompt:

"<think>\n"

The model was found to be highly capable on reasoning tasks when skipping think block, with low overthinking, just accurate direct deductions to final solutions. On ambiguous/trick prompts the model can generate very long reasoning blocks in think mode, but non ambiguous prompts are normally concise, accurate and efficient.

VISION:

The quant was tested in vision mode on a couple pretty tough bird ID image and did well, with overthinking on one of the prompts but accurate final answer.

CODE:

The model was tested across a small set of code gen prompts and found to be good in its ability to generate working code with or without think block enabled, generating working code for the majority of the prompts. On one prompt it got into a rep loop with think mode enabled but forcing think block off stopped the rep loop.

This version of the model needs a verion of llama.cpp which supports MTP due to included MTP layer; if not using MTP warning messages will be generated about unused tensors but the model will run.

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3.5-35B-A3B.Q4_E_H.gguf	Q4_E_H	21.8e9 B	~Q4_K_M size, includes ~ 0.4B MPT
Qwen3.5-35B-A3B.mmproj.gguf	F16	0.90e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040