YOYO-AI/Qwen3-30B-A3B-YOYO-V5

The 5th-Generation Merged Model of YOYO-AI and the brand-new merging algorithm "yoyo_fusion" have been officially released!

Model Highlights:

• merge method: yoyo_fusion

• precision: dtype: bfloat16

• Context length: 262,144&1010000

Parameter Settings:

Temperature=0.7, TopP=0.8, TopK=20,MinP=0.

GitHub Repository:

YOYO-Fusion

Configuration:

The following configuration was used to produce this model:

from yoyo_fusion import run_merge

run_merge(
    model_paths=[
        "Qwen/Qwen3-30B-A3B-Instruct-2507",
        "Qwen/Qwen3-30B-A3B-Thinking-2507",
        "Qwen/Qwen3-Coder-30B-A3B-Instruct"
    ],
    output_dir="YOYO-AI/Qwen3-30B-A3B-YOYO-V5",
    anchor_index=0,
    config_dir=1,
    use_k_minus_one_truncation=True,
    use_geometric_median=True,
)

YOYO-Fusion: Robust Merging in Residual Subspace

Input

Given K≥2 weight tensors from models with identical architecture:
$$\{T^{(1)},T^{(2)},\dots ,T^{(K)}\},T^{(k)}\in {\mathbb{R}}^{d_1\times \cdots \times d_n},\backslash \{T^\{(1)\},\; T^\{(2)\},\; \backslash dots,\; T^\{(K)\}\backslash \},\; \backslash quad\; T^\{(k)\}\; \backslash in\; \backslash mathbb\{R\}^\{d\_1\; \backslash times\; \backslash cdots\; \backslash times\; d\_n\},$$

---

Step 1: Flatten and RMS-normalize each tensor

Flatten each tensor into a vector and normalize by its RMS:
$$x^{(k)}=\mathrm{flatten}(T^{(k)})\in {\mathbb{R}}^D,D=\prod _{i=1}^n{d}_{i}x^\{(k)\}\; =\; \backslash operatorname\{flatten\}(T^\{(k)\})\; \backslash in\; \backslash mathbb\{R\}^D,\; \backslash quad\; D\; =\; \backslash prod\_\{i=1\}^n\; d\_i$$
$$r_k=\mathrm{RMS}(x^{(k)})=\sqrt{\frac{1}{D}\sum _{i=1}^D(x_i^{(k)}{)}^2+\epsilon }r\_k\; =\; \backslash operatorname\{RMS\}(x^\{(k)\})\; =\; \backslash sqrt\{\; \backslash frac\{1\}\{D\}\; \backslash sum\_\{i=1\}^D\; (x^\{(k)\}\_i)^2\; +\; \backslash varepsilon\; \}$$
$$u^{(k)}=\frac{x^{(k)}}{r_k+\epsilon }u^\{(k)\}\; =\; \backslash frac\{x^\{(k)\}\}\{r\_k\; +\; \backslash varepsilon\}$$

---

Step 2: Determine Center Point

Case A: Anchor Mode

$$\mathbf{m}={\mathbf{u}}_n\backslash mathbf\{m\}\; =\; \backslash mathbf\{u\}\_n$$

Case B: No Anchor Mode

• Subcase B1:

  Compute the geometric median via the Weiszfeld algorithm:

$$\mathbf{m}=\arg \underset{\mathbf{y}}{\min }\sum _{i=1}^K\mathrm{\parallel }{\mathbf{u}}_i-\mathbf{y}{\mathrm{\parallel }}_2\backslash mathbf\{m\}\; =\; \backslash arg\backslash min\_\{\backslash mathbf\{y\}\}\; \backslash sum\_\{i=1\}^K\; \backslash |\; \backslash mathbf\{u\}\_i\; -\; \backslash mathbf\{y\}\; \backslash |\_2$$

• Subcase B2:

  Use coordinate-wise median:

$$m_j=\text{median}(u_{1,j},u_{2,j},\dots ,u_{K,j}),\mathrm{\forall }j=1,\dots ,Dm\_j\; =\; \backslash text\{median\}(u\_\{1,j\},\; u\_\{2,j\},\; \backslash dots,\; u\_\{K,j\}),\; \backslash quad\; \backslash forall\; j=1,\backslash dots,D$$

---

Step 3: Compute residual matrix

$$\mathbf{R}=\mathbf{U}-{\mathbf{1}}_K{\mathbf{m}}^{\mathrm{\top }}\in {\mathbb{R}}^{K\times D}\backslash mathbf\{R\}\; =\; \backslash mathbf\{U\}\; -\; \backslash mathbf\{1\}\_K\; \backslash mathbf\{m\}^\backslash top\; \backslash in\; \backslash mathbb\{R\}^\{K\; \backslash times\; D\}$$

---

Step 4: Early exit if residuals are negligible

If

then set
$${\mathbf{y}}^{\mathrm{\prime }}=\mathbf{m}\backslash mathbf\{y\}\text{'}\; =\; \backslash mathbf\{m\}$$
and skip to Step 8. Otherwise, proceed.

---

Step 5: Perform SVD on residuals

Compute the thin SVD of R^⊤∈R^D×K:
$$R^{\mathrm{\top }}=U\mathrm{\Sigma }{V}^{\mathrm{\top }}R^\backslash top\; =\; U\; \backslash Sigma\; V^\backslash top$$
Let min(K−1,rank(R)), and take the first r' columns of U :
$$U_{r^{\mathrm{\prime }}}=U[:,:r^{\mathrm{\prime }}]\in {\mathbb{R}}^{D\times r^{\mathrm{\prime }}}U\_\{r\text{'}\}\; =\; U[:,\; :r\text{'}]\; \backslash in\; \backslash mathbb\{R\}^\{D\; \backslash times\; r\text{'}\}$$

---

Step 6: Compute energy-based scaling factor

Total energy:
$$E_{\text{total}}=\sum _{i=1}^{\mathrm{rank}}{\sigma }_i^{2}E\_\{\backslash text\{total\}\}\; =\; \backslash sum\_\{i=1\}^\{\backslash operatorname\{rank\}\}\; \backslash sigma\_i^2$$
Retained energy:
$$E_{\text{retained}}=\sum _{i=1}^{r^{\mathrm{\prime }}}{\sigma }_i^{2}E\_\{\backslash text\{retained\}\}\; =\; \backslash sum\_\{i=1\}^\{r\text{'}\}\; \backslash sigma\_i^2$$
Energy ratio:
$$p=\frac{E_{\text{retained}}}{E_{\text{total}}+\epsilon }p\; =\; \backslash frac\{E\_\{\backslash text\{retained\}\}\}\{E\_\{\backslash text\{total\}\}\; +\; \backslash varepsilon\}$$
Scaling factor (clamped for stability):
$$\lambda =\min (\frac{1}{p+\epsilon },10.0)\backslash lambda\; =\; \backslash min\backslash left(\; \backslash frac\{1\}\{p\; +\; \backslash varepsilon\},\backslash \; 10.0\; \backslash right)$$

---

Step 7: Robust weighted averaging in subspace

Project residuals into subspace

$$Z=R{U}_{r^{\mathrm{\prime }}}\in {\mathbb{R}}^{K\times r^{\mathrm{\prime }}}Z\; =\; R\; U\_\{r\text{'}\}\; \backslash in\; \backslash mathbb\{R\}^\{K\; \backslash times\; r\text{'}\}$$

Estimate robust scales

Per-coordinate MAD scale:
$$s_j=1.4826\cdot {\mathrm{median}}_k\left(\mathrm{\mid }{Z}_{k,j}\mathrm{\mid }\right),j=1,\dots ,r^{\mathrm{\prime }}s\_j\; =\; 1.4826\; \backslash cdot\; \backslash operatorname\{median\}\_\{k\}\; \backslash left(\; |Z\_\{k,j\}|\; \backslash right),\; \backslash quad\; j\; =\; 1,\; \backslash dots,\; r\text{'}$$
Per-model residual norm:
$$\mathrm{\parallel }{z}_k\mathrm{\parallel }=\mathrm{\parallel }{Z}_{k,:}{\mathrm{\parallel }}_2\backslash |z\_k\backslash |\; =\; \backslash |Z\_\{k,:\}\backslash |\_2$$
Global MAD scale:
$$s_{\text{global}}=1.4826\cdot {\mathrm{median}}_k\left(\mathrm{\parallel }{z}_k\mathrm{\parallel }\right)s\_\{\backslash text\{global\}\}\; =\; 1.4826\; \backslash cdot\; \backslash operatorname\{median\}\_\{k\}\; \backslash left(\; \backslash |z\_k\backslash |\; \backslash right)$$

Compute Tukey bisquare weights（c = 4.685）

Coordinate-wise weights:
$$w_{k,j}^{\text{coord}}={[\max (0,1-{\left(\frac{\mathrm{\mid }{Z}_{k,j}\mathrm{\mid }}{c\cdot s_j+\epsilon }\right)}^2)]}^{2}w^\{\backslash text\{coord\}\}\_\{k,j\}\; =\; \backslash left[\; \backslash max\backslash left(\; 0,\backslash \; 1\; -\; \backslash left(\; \backslash frac\{|Z\_\{k,j\}|\}\{c\; \backslash cdot\; s\_j\; +\; \backslash varepsilon\}\; \backslash right)^2\; \backslash right)\; \backslash right]^2$$
Global (per-model) weights:
$$w_k^{\text{global}}={[\max (0,1-{\left(\frac{\mathrm{\parallel }{z}_k\mathrm{\parallel }}{c\cdot s_{\text{global}}+\epsilon }\right)}^2)]}^{2}w^\{\backslash text\{global\}\}\_k\; =\; \backslash left[\; \backslash max\backslash left(\; 0,\backslash \; 1\; -\; \backslash left(\; \backslash frac\{\backslash |z\_k\backslash |\}\{c\; \backslash cdot\; s\_\{\backslash text\{global\}\}\; +\; \backslash varepsilon\}\; \backslash right)^2\; \backslash right)\; \backslash right]^2$$
Combined weights:
$$W_{k,j}=w_{k,j}^{\text{coord}}\cdot w_k^{\text{global}}W\_\{k,j\}\; =\; w^\{\backslash text\{coord\}\}\_\{k,j\}\; \backslash cdot\; w^\{\backslash text\{global\}\}\_k$$

Compute robust consensus in subspace

$$z_j^{\ast }=\frac{\sum _{k=1}^K{W}_{k,j}{Z}_{k,j}}{\sum _{k=1}^K{W}_{k,j}+\epsilon },j=1,\dots ,r^{\mathrm{\prime }}z^*\_j\; =\; \backslash frac\{\; \backslash sum\_\{k=1\}^K\; W\_\{k,j\}\; Z\_\{k,j\}\; \}\{\; \backslash sum\_\{k=1\}^K\; W\_\{k,j\}\; +\; \backslash varepsilon\; \},\; \backslash quad\; j\; =\; 1,\; \backslash dots,\; r\text{'}$$
Reconstruct robust residual:
$$r^{\ast }=\lambda \cdot U_{r^{\mathrm{\prime }}}{z}^{\ast }\in {\mathbb{R}}^{D}r^*\; =\; \backslash lambda\; \backslash cdot\; U\_\{r\text{'}\}\; z^*\; \backslash in\; \backslash mathbb\{R\}^D$$
Final estimate in normalized space:
$$y^{\mathrm{\prime }}=m+r^{\ast }y\text{'}\; =\; m\; +\; r^*$$

---

Step 8: Restore average RMS scale

Compute mean RMS across inputs:
$$\bar{r}=\frac{1}{K}\sum _{k=1}^K{r}_k\backslash bar\{r\}\; =\; \backslash frac\{1\}\{K\}\; \backslash sum\_\{k=1\}^K\; r\_k$$
Scale back:
$$y=y^{\mathrm{\prime }}\cdot \bar{r}y\; =\; y\text{'}\; \backslash cdot\; \backslash bar\{r\}$$

---

Step 9: Final L2 norm alignment

Compute average L2 norm of original flattened tensors:
$$\bar{n}=\frac{1}{K}\sum _{k=1}^K\mathrm{\parallel }{x}^{(k)}{\mathrm{\parallel }}_2\backslash bar\{n\}\; =\; \backslash frac\{1\}\{K\}\; \backslash sum\_\{k=1\}^K\; \backslash |x^\{(k)\}\backslash |\_2$$
Compute current norm:
$$n_y=\mathrm{\parallel }y{\mathrm{\parallel }}_{2}n\_y\; =\; \backslash |y\backslash |\_2$$
Final scaling factor:
$$\alpha =\frac{\bar{n}}{n_y+\epsilon }\backslash alpha\; =\; \backslash frac\{\backslash bar\{n\}\}\{n\_y\; +\; \backslash varepsilon\}$$
Scaled output vector:
$$\widehat{x}=\alpha \cdot y\backslash hat\{x\}\; =\; \backslash alpha\; \backslash cdot\; y$$
Reshape to original tensor shape:
$$\widehat{T}=\mathrm{reshape}(\widehat{x},(d_1,\dots ,d_n))\backslash hat\{T\}\; =\; \backslash operatorname\{reshape\}(\backslash hat\{x\},\backslash \; (d\_1,\; \backslash dots,\; d\_n))$$