Add MLX 4-bit model card

README.md

@@ -0,0 +1,178 @@
+---
+license: mit
+base_model: inclusionAI/Ling-3.0-flash
+library_name: mlx
+pipeline_tag: text-generation
+tags:
+  - mlx
+  - mlx-lm
+  - omlx
+  - bailing_hybrid
+  - mixture-of-experts
+  - quantized
+  - 4-bit
+---
+
+<p align="center">
+  <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100" alt="Ling logo">
+</p>
+
+<h1 align="center">Ling 3.0 Flash — MLX 4-bit</h1>
+
+<p align="center">
+  A native Apple-silicon conversion of <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash">inclusionAI/Ling-3.0-flash</a>, quantized with the stock MLX-LM converter and packaged for MLX-LM and oMLX.
+</p>
+
+<p align="center">
+  <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash">Original model</a> ·
+  <a href="https://modelscope.cn/organization/inclusionAI">InclusionAI on ModelScope</a> ·
+  <a href="https://openrouter.ai/inclusionai/ling-3.0-flash:free">OpenRouter</a>
+</p>
+
+## About this conversion
+
+This repository contains a **4-bit affine MLX conversion** of Ling 3.0 Flash. Ling is a 124B-parameter hybrid-linear mixture-of-experts model with approximately 5.1B active parameters per token. The conversion preserves the upstream tokenizer and chat template and includes an MLX architecture adapter for the model's KDA, MLA, and sparse-MoE layers.
+
+| Item | Value |
+| --- | --- |
+| Base model | [`inclusionAI/Ling-3.0-flash`](https://huggingface.co/inclusionAI/Ling-3.0-flash) |
+| Format | MLX safetensors |
+| Quantization | 4-bit affine, group size 64 |
+| Converter | Stock `mlx_lm.convert` from MLX-LM 0.31.3 |
+| Effective weight precision | 4.502 bits per weight |
+| Repository size | 70.03 GB (65.22 GiB) |
+| Maximum configured context | 262,144 tokens |
+| Architecture | `bailing_hybrid` |
+
+> [!IMPORTANT]
+> This model includes the custom architecture file `bailing_hybrid.py`. Review the file and enable **Trust Remote Code** before loading it. In oMLX, enable Trust Remote Code in the model settings. With MLX-LM, pass `--trust-remote-code`.
+
+## Apple-silicon validation
+
+The converted weights were loaded strictly with both stock MLX-LM and the MLX runtime bundled with oMLX. End-to-end generation was also tested through the oMLX OpenAI-compatible API on an Apple-silicon Mac Studio.
+
+One short local smoke test produced:
+
+| Measurement | Result |
+| --- | ---: |
+| Prefill | 21.38 tokens/s |
+| Decode | 76.41 tokens/s |
+| Reported peak memory | 70.18 GB |
+| oMLX API | HTTP 200, valid completion |
+
+These figures confirm that the model runs; they are not a controlled benchmark. Prompt length, sampler settings, macOS memory pressure, thermal state, and MLX/oMLX versions can materially change performance.
+
+## Quick start with MLX-LM
+
+Install recent MLX-LM and Hugging Face tooling:
+
+```bash
+python -m pip install -U mlx-lm huggingface_hub
+```
+
+Run a chat prompt directly from the Hub:
+
+```bash
+mlx_lm.generate \
+  --model Vontra/Ling-3.0-flash-MLX-4bit \
+  --trust-remote-code \
+  --prompt "Explain why hybrid linear attention is useful." \
+  --max-tokens 512 \
+  --temp 0.6 \
+  --top-p 0.95 \
+  --top-k 20
+```
+
+Thinking mode is enabled by the upstream chat template by default. It can be disabled with:
+
+```bash
+mlx_lm.generate \
+  --model Vontra/Ling-3.0-flash-MLX-4bit \
+  --trust-remote-code \
+  --chat-template-config '{"enable_thinking": false}' \
+  --prompt "Write a short hello-world program in Swift."
+```
+
+To download the repository first:
+
+```bash
+hf download Vontra/Ling-3.0-flash-MLX-4bit \
+  --local-dir ~/.omlx/models/Vontra/Ling-3.0-flash-MLX-4bit
+```
+
+## Using it with oMLX
+
+1. Place the model at `~/.omlx/models/Vontra/Ling-3.0-flash-MLX-4bit`.
+2. Refresh the oMLX model registry.
+3. Open the model settings and enable **Trust Remote Code**.
+4. Load `Ling-3.0-flash-MLX-4bit` and use the normal chat or OpenAI-compatible endpoint.
+
+Example request:
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $OMLX_API_KEY" \
+  -d '{
+    "model": "Ling-3.0-flash-MLX-4bit",
+    "messages": [{"role": "user", "content": "Say hello from MLX."}],
+    "temperature": 0.6,
+    "top_p": 0.95,
+    "max_tokens": 128
+  }'
+```
+
+For long prompts, begin with a conservative context limit and increase it while watching memory pressure. The advertised 256K context is a model capability, not a promise that every host can prefill that context within its available unified memory.
+
+## Architecture
+
+Ling 3.0 Flash is a native hybrid-linear MoE model. It alternates Kimi Delta Attention (KDA) and gated Multi-head Latent Attention (MLA) in a 5:1 ratio and uses highly sparse routed experts.
+
+| Architecture detail | Upstream value |
+| --- | ---: |
+| Total / active parameters | 124B / 5.1B |
+| Transformer layers | 35 KDA + 7 gated MLA |
+| Dense layers | 2 |
+| Routed / shared experts | 512 / 1 |
+| Active routed experts | 8 |
+| Attention heads | 32 |
+| Hidden size | 2,560 |
+| Expert intermediate size | 768 |
+| Dense intermediate size | 6,144 |
+| Vocabulary size | 157,184 |
+| Context training schedule | 8K → 32K → 256K |
+
+The included MLX adapter uses MLX-LM's native primitives for delta attention, absorbed MLA projections, RoPE, and quantized SwitchGLU experts. The auxiliary MTP training head is intentionally excluded from ordinary causal generation; this release does **not** claim MTP/DSpark speculative decoding support.
+
+## Upstream model highlights
+
+InclusionAI describes Ling 3.0 Flash as a next-generation hybrid reasoning model designed for efficient long-context inference and agentic workloads. The upstream release reports strong results across software-engineering agents, tool use, deep research, general knowledge, mathematical reasoning, instruction following, and long-context understanding.
+
+The upstream defaults are:
+
+```text
+thinking: enabled
+temperature: 0.6
+top_p: 0.95
+top_k: 20
+```
+
+For benchmark methodology, scores, limitations, and framework-specific SGLang/vLLM deployment instructions, see the [original InclusionAI model card](https://huggingface.co/inclusionAI/Ling-3.0-flash).
+
+## Conversion notes
+
+- Source weights: BF16 upstream checkpoint.
+- Conversion command: stock `mlx_lm.convert` with `--quantize --q-bits 4 --q-group-size 64 --q-mode affine`.
+- The upstream `chat_template.jinja` is included unchanged.
+- All source tensors required for base causal generation were consumed by a strict load before conversion.
+- The 42 base transformer layers are retained. The separate depth-5 MTP training/speculation stack is not part of this runtime adapter.
+
+Four-bit weight quantization materially reduces memory use but may reduce output quality relative to BF16 or the 8-bit conversion.
+
+This is a community conversion, not an official InclusionAI release. Because the architecture adapter is new, validate quality and numerical behavior on your own representative workload before production use.
+
+## License and attribution
+
+The upstream model is released under the **MIT License**. This conversion preserves that license and is derived from [`inclusionAI/Ling-3.0-flash`](https://huggingface.co/inclusionAI/Ling-3.0-flash).
+
+All model design, training, and benchmark credit belongs to InclusionAI and the original contributors. The MLX conversion and compatibility adapter are provided by [Vontra](https://huggingface.co/Vontra).