Where do you spend 100M adapter parameters on a looped transformer?

Most fine-tuning research assumes a stacked transformer: N distinct layers, each with its own weights. But a growing class of models — looped transformers — reuse the same FFN block across multiple forward passes. That changes the parameter-allocation question in a way the literature hadn't settled.

We had a ~100M parameter budget for an adapter on chili-lab/Ouro-hybrid-1.4B, a looped transformer with four parameter-sharing passes. Spread the budget across every FFN sublayer as a multiplicative gate, or concentrate it in the output projection? We ran the same budget both ways across five experiments. Distributing beat concentrating by 19–84% every single time.

What a looped transformer is

A standard transformer has N distinct layers stacked on top of each other. Each layer has its own weights — its own attention, its own FFN. Compute flows bottom to top, once.

A looped transformer factors the model differently. It has fewer distinct FFN blocks, but each block is run K times — the same FFN weights participate at multiple depths of the computation. chili-lab/Ouro-hybrid-1.4B, the base model we worked with, does four such passes. It's a 1.4B-parameter model that behaves computationally like something deeper.

This structure creates an asymmetry the stacked-transformer literature doesn't have to reckon with. When you fine-tune, the question is no longer "which layer do the adapter params live on?" — because a single FFN block is every layer. The question is "how does the adapter interact with each pass through that block?" And that's a different question entirely.

---

The experiment

We compared two ways to spend a fixed ~100M parameter budget on the same base model:

• Baseline (concentrated). Put all 100M trainable params in the output projection (lm_head). This is the standard cheap-and-effective way to adapt an LLM — you get to reshape the vocabulary distribution at the top of the stack.
• Gate (distributed). Insert a multiplicative gate at every FFN sublayer, at every loop depth: h = h + relu(gate_proj(ffn_norm(h))) * feed_forward(ffn_norm(h)). The gate learns to scale the FFN's contribution differently at each pass. Same 100M-param budget, spread across the model instead of concentrated at the top.

Base model: chili-lab/Ouro-hybrid-1.4B (itself finetuned from ByteDance/Ouro-1.4B). Data: GSM8K train split, 7,473 Q/A pairs. Optimizer: AdamW, cosine LR decay 1e-3 → 5e-5. Hardware: single RTX 5080, batch size 1 × seq_len 256. Pure PyTorch sequential recurrence, no fancy kernels. This ran on a desk.

Five experiments across two domains and three loss regimes. Same base model, same budget, only the adapter placement changed between arms.

---

The result

Distributed won every experiment. Not close ones.

Metric (5,000 steps on GSM8K, cosine LR)	Gate (distributed)	lm_head (concentrated)
Validation loss before → after (nats)	8.74 → 1.57	8.74 → 4.35
Improvement (nats)	+7.11	+4.44
Advantage at identical param budget	60% more perplexity improvement

Across the full five-experiment sweep, the gate's advantage ranged from 19% to 84%. The floor was still a decisive win. The ceiling was nearly double the baseline's improvement at the same cost.

Reading this back it looks like an obvious result — of course spreading params across every pass helps a model that's making multiple passes. But this hadn't been shown for looped transformers, and the naïve prior (from stacked-transformer intuition) is the opposite: concentrate params where the output leaves the model, don't waste them in the middle. Looped structure inverts that. What looks like "the middle" in a stacked transformer is actually every pass of the same block in a looped one, and every pass is worth shaping.

---

Identity init is doing more work than it looks

One detail was load-bearing enough that we want to call it out: the gate must be initialised at identity — W_g = 0, b_g = 1 — so the patched model is bit-identical to the pretrained baseline at step 0.

Random initialisation didn't just underperform. It actively degraded performance, and the degradation compounded with loop depth. The deeper the loop, the worse random init got. This makes sense once you see it: if you insert a modification at every pass through a shared block, K noisy initialisations compound instead of cancelling. Training then has to spend most of its early steps undoing the noise before it can start adding signal.

Identity init sidesteps this entirely. The model at step 0 is exactly the pretrained model. Every gradient step is additive signal, not noise correction. In a stacked transformer this detail matters at the margin. In a looped transformer it's the difference between the method working and not.

---

The honest asterisk

Perplexity dropped by 7+ nats. GSM8K exact-match accuracy across all adapter configurations, gate and baseline both, is zero.

The gate visibly acquired target-domain register, format, and chain-of-thought structure — the outputs look like GSM8K solutions. What they don't do is arrive at the correct number. Something the loop structure is doing (or not doing) at this scale isn't captured by the perplexity signal, and closing the gap between "sounds right" and "gets it right" is the primary open question.

We say this out loud because it matters. The finding is "depth-wise adapter placement outperforms output-layer concentration by 19–84% in perplexity at identical budget." The finding is not "we solved GSM8K with a 100M adapter." Anyone reading this and wondering whether to spend their compute on gate adapters should know the second sentence exists.

---

Try it

Model weights and full usage code are on Hugging Face: YG3-ai/dragonling-gate-adapter. Sketch:

from bdh_lt2.student_model import register_student_model
from bdh_lt2.gating import apply_bdh_gating
from transformers import AutoModelForCausalLM, AutoTokenizer
from safetensors.torch import load_file

register_student_model()
model = AutoModelForCausalLM.from_pretrained(
    "chili-lab/Ouro-hybrid-1.4B",
    torch_dtype="auto", device_map="auto", trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "chili-lab/Ouro-hybrid-1.4B", trust_remote_code=True,
)

apply_bdh_gating(model, init_identity=True)
gate_sd = load_file("adapter_weights.safetensors")
model.load_state_dict(gate_sd, strict=False)

Full paper on Zenodo: Multiplicative Gate Adapters for Looped Transformers (CC-BY-4.0, DOI 10.5281/zenodo.21311787). Adapter weights are released under the chili-lab research license, consistent with the base model.

---

What's open

• Perplexity → task accuracy transfer. The primary open question. Does the register / format / chain-of-thought structure the gate learns translate to task accuracy at larger scales, longer training, or different task families? Or does looped structure need something the current gate formulation isn't providing?
• Where else does this apply? We tested one base model with four parameter-sharing passes. Whether depth-wise distribution generalises across loop depths, model sizes, and base architectures is open.
• Efficient implementation. The current recurrence is a plain PyTorch loop for legibility. A fused-kernel implementation should be a straightforward win for anyone who wants to train these at scale.

If you're working on looped transformers or PEFT and want to compare notes: research@yg3.ai.

Paper by Jacqueline Carter and Sam Knox at YG3. Full paper on Zenodo, adapter weights on Hugging Face.