Part I: Topology, Input Injection, Recurrent-State Design

Benhao Huang‡ †, Chufan Shi‡, Junlin Chen‡, Shicheng Wen‡,

Zhengzhong Liu‡, Eric Xing‡, Xuezhe Ma‡*****

*‡Institute of Foundation Models, USC, †CMU

Date: July 31, 2026

Fig. 1 Parameter scaling and benchmark performance of loop and feedforward MoE models. Left: Ouro MoE and Huginn MoE share the same scale: 8.0B resident and 0.8B active parameters, corresponding to 32.0B resident-equivalent and 3.2B unrolled-active parameter applications. Unrolled counts measure parameter applications under weight reuse. Right: Huginn MoE outperforms Ouro MoE overall, and approaches or surpasses the 112-layer feedforward MoE baseline on DROP, MATH500 and GSM8K, while remaining competitive on the other benchmarks. All models use the same data and matched training and inference FLOPs, while looped models require less memory.

†Correspondence to: Benhao Huang [email protected]

Code (Release Soon) | Living Blog (updating continuously)

• Cite this work

<aside> 💡

TL;DR

Loop language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization. We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test whether the resulting design principles transfer to resident- and active-parameter-matched Mixture-of-Experts (MoE) models.

• Loop design. The prelude–recurrent block–coda sandwich architecture consistently improves performance on context extraction and reasoning-intensive tasks. Input injection primarily enhances knowledge retrieval and context utilization, whereas random initialization and separate H/L recurrent states provide mixed or negative returns. Taken together, these results favor the Huginn-style design: at the 730M-parameter and 336B-token scale, the complete Huginn architecture outperforms Ouro across all ten benchmarks.
• MoE transfer. The advantages of the Huginn over Ouro transfer to Mixture-of-Experts models. At the 8BA0.8B (32BA3B when unrolled) scale trained on 500B tokens, Huginn outperforms Ouro nearly all benchmarks spanning knowledge, reasoning, and coding, while also exhibiting more balanced expert routing.
• Apples-to-apples comparison. Under matched training and inference FLOPs, the Huginn-style model uses 75% fewer resident parameters than a 112-layer feedforward MoE baseline. Despite its smaller physical parameters, it outperforms the feedforward baseline on DROP and GSM8K, matches it on MATH500, and remains competitive on the other benchmarks. </aside>

Tab. 1 Benchmark performance of loop MoE models and the feedforward MoE reference after training on 500B tokens. The looped models contain 8.0B resident and 793.9M active parameters per physical pass, corresponding to approximately 630 training tokens per active parameter. Bold compares only the recurrent rows; $\Delta$ is Huginn MoE minus Ouro MoE; green and red mark positive and negative changes, respectively.

Table of Content

1. Introduction

Loop language models repeatedly reuse the same Transformer blocks across logical depth, trading additional computation for reduced memory usage. As modern AI hardware delivers rapidly increasing compute throughput while memory capacity and bandwidth improve much more slowly [1], this paradigm has emerged as a promising direction for scaling language models. Recent work has investigated recurrent-depth scaling [2–5], capability profiles [6-9], as well as the roles of stability and residual dynamics [8,17], architecture topology and sparsity [10,18], and memory efficiency [11]. Most existing loop language models can be broadly categorized into two architectural lineages, represented by Ouro [2] and Huginn [3].

Despite their growing influence, the architectural differences between these two families have not been systematically isolated. Ouro recurrently applies a tied full stack initialized from token embeddings, whereas Huginn places recurrence between untied prelude and coda layers, repeatedly injects a fixed prelude representation, and uses a separately initialized recurrent state. We formalize these differences along three axes---the iteration envelope, input injection, and latent-state organization---and isolate their effects through a controlled Ouro-to-Huginn transformation. We pretrain all resulting models from scratch under matched parameter scale, logical depth, and token budgets, and evaluate them using a common protocol.

Fig. 2 Selected loop-model works organized by family and quarter.

The rest of this article follows three questions.