Instructions to use google/diffusiongemma-26B-A4B-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/diffusiongemma-26B-A4B-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/diffusiongemma-26B-A4B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("google/diffusiongemma-26B-A4B-it")
model = AutoModelForMultimodalLM.from_pretrained("google/diffusiongemma-26B-A4B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/diffusiongemma-26B-A4B-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/diffusiongemma-26B-A4B-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/diffusiongemma-26B-A4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/google/diffusiongemma-26B-A4B-it

SGLang

How to use google/diffusiongemma-26B-A4B-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/diffusiongemma-26B-A4B-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/diffusiongemma-26B-A4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/diffusiongemma-26B-A4B-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/diffusiongemma-26B-A4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use google/diffusiongemma-26B-A4B-it with Docker Model Runner:
```
docker model run hf.co/google/diffusiongemma-26B-A4B-it
```

Field report: DiffusionGemma 26B as a full Claude Code agent on a single RTX 5090 (WSL2) — what worked, and the guardrails for the accuracy problem

#21

by Markbdean - opened 11 days ago

Discussion

Markbdean

11 days ago

I'm a retired builder/hacker in the UK who tinkers with agentic systems for fun, not notoriety — but a few things I hit running DiffusionGemma as a real agent might save people an evening, so here's a brain-dump. Happy to be wrong on any of it.

Setup: DiffusionGemma 26B-A4B (NVFP4) on vLLM, wired up as a full Claude Code agent — ANTHROPIC_BASE_URL pointed straight at vLLM's native /v1/messages endpoint, no shim, --enable-auto-tool-choice --tool-call-parser gemma4. Consumer RTX 5090 32GB, WSL2 on Windows 11. It runs my actual home-automation lanes — email/calendar, a hybrid RAG knowledge base, web research, browser automation, task registers — ~11 genuine multi-step tool scenarios, all passing, typically 10–20s end-to-end.

A few things I learned that might be useful:

Prompt size dominates diffusion latency far more than I expected (coming from autoregressive models). Trimming the client system prompt from ~24.5k to ~2.6k tokens — a custom agent definition with 5 tools instead of the stock ~27 — took one email task from ~5 min to ~10s. If you're wiring an agent client to this model: trim the tool schemas first, tune the sampler second.
Aggressive sampler tuning backfired — pushing the entropy bound / denoising steps made answers worse and slower end-to-end (worse answers mean more agent steps).
The accuracy problem is real, and the fix is architecture, not the model. It's fast but it hallucinates names/commands more than an AR model — risky to let it run bash autonomously. What works for me: a deterministic "fact-critical wall" (the tool does the action/read; the model only phrases the result) plus a cascade — the local model handles the volume and self-escalates to a frontier model for the hard calls. That turns "4× faster but error-prone" into something you can actually trust in the loop.
Restart anatomy on a warm 5090: weights 30s, torch.compile 5s (AOT cache hit — nice), flashinfer autotune ~82s (re-runs every start), warmup/CUDA-graph ~75s → ~3m40s to serving.

Not selling anything — just closing the loop with real-workload data, and honestly hoping to find anyone else doing training (LoRA / diffu-GRPO) or agentic-harness work on this thing, because that side seems wide open. If that's you, say hi.

(Written up with my AI sidekick, but every number above is from my own rig.)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment