What If a Transformer Never Had to Forget? Meet the Recurrent Looped Transformer (RLT)

You ask a language model a one-line question — it processes it through 48 layers. You paste a 10,000-word document — it still processes it through 48 layers. Same depth. Same ceiling. Every single time.

That's the structural limit of every decoder-only Transformer in production today.

A technical report published September 12, 2026 by Princeton researcher Yifan Zhang — Recurrent Looped Transformer (RLT) — proposes closing that loop. In most decoder-only LLMs, nothing computed at the last layer of token t feeds the first layer of token t+1; positions communicate only through attention over cached keys and values. RLT changes this: the decoder's final hidden state and its layerwise sliding-window attention (SWA) cache are carried into the next token, across both prompt and response, with no reset at the boundary.

⚠️ Important: RLT is an architectural specification, not a trained system. The report explicitly states that no measured efficiency, reasoning quality, or scaling results are reported.

How RLT Is Built

Recurrent Looped Transformer (RLT) pairs a causal encoder with a recurrent decoder.

Causal Encoder

The encoder processes tokens in parallel under a causal mask and produces representations for each position. These representations are projected into key-value memory that the decoder can query later. Memory groups can be:

• Shared across all decoder layers (efficient), or
• Layer-specific (more flexible)

This memory is encoder-derived — it depends only on input tokens, not on decoder states.

Recurrent Decoder

The decoder holds the recurrence. Its complete state has two components:

1. Final decoder output (the recurrent hidden state)
2. Sliding-window attention (SWA) cache — retained keys and values at every decoder layer

For each token, a gated merge combines the current encoder representation with the previous decoder output. The gate controls how much previous information is carried forward. Then each decoder block runs:

1. Causal SWA over decoder activations (local attention within a bounded window)
2. Cross-attention to encoder memory (global context)
3. Feed-forward network (standard FFN)

The next-token distribution is read from the final decoder output. Initialization happens once before the beginning-of-sequence token with a learned start state and an empty cache.

💡 Tip: RLT separates two memory stores because they serve different roles:

• Encoder memory is immutable for a fixed prefix — think of it as a "textbook index" of what was said
• Decoder SWA cache changes every step and keeps only recent activations — think of it as the "last 2 pages of notes"

Neither can replace the other.

Reference Tied Configuration

The reference configuration uses:

• 48 encoder layers
• 48 decoder layers
• Compatible attention and FFN weights shared between them

Each token executes 96 logical blocks (48 encoder + 48 decoder). Zhang calls this parameter reuse, not activation copying. Two logical passes do not imply equal per-block compute — the decoder block adds cross-attention that the encoder doesn't have.

The 3 Design Principles

1. Latent Reasoning with Unbounded Temporal Depth

After processing t tokens, the state path traverses t × 48 decoder blocks in the reference configuration.

Per-token work stays fixed at 96 blocks, while the path's structural depth grows with the sequence. "Unbounded" means no fixed architectural upper bound on the recurrent computation path — it does not mean infinite computation per input.

The report warns that gates and contraction may suppress long paths. Structural depth is not a reasoning guarantee — it's a structural possibility that requires experimental validation.

💡 Tip — Snowball Analogy: The same amount of snow is added during every rotation (fixed compute per token), but the snowball grows because information accumulates. After 100 rotations, you have a massive snowball — without needing a bigger hill.

2. Model–Hardware Co-Design

Encoder features use token-parallel kernels — they can process known tokens in parallel. Decoder transitions stay sequential within a sequence, but independent sequences can be batched together:

Sequence A: Token 1 → Token 2 → Token 3 → Token 4
Sequence B: Token 1 → Token 2 → Token 3 → Token 4
Sequence C: Token 1 → Token 2 → Token 3 → Token 4
↓
BATCHED ON GPU

The report states plainly:

• No exact parallel scan is assumed for the nonlinear decoder
• No reduced-prefill speedup is claimed
• A standard parallel SWA decoder pass is not equivalent to the recurrence
• Batching, kernel fusion, and checkpointing are listed as implementation targets, not completed kernels

💡 Tip for Practitioners: RLT's decoder is inherently sequential per sequence. Throughput optimization comes from batching independent sequences, not from parallelizing within one sequence. This is closer to RNN-style serving than standard transformer serving.

3. Model–RL Algorithm Co-Design

Pretraining, supervised fine-tuning (SFT), sampling, and reinforcement learning (RL) replay share one state transition.

During sampling:

• Record generated tokens
• Record behavior log-probabilities under the actual sampling distribution (including temperature and truncation)
• Record sampling configuration

During training:

• Rebuild encoder memory from scratch under current parameters
• Rebuild the recurrent output through all prompt tokens under current parameters
• Rebuild every SWA cache under current parameters
• Old rollout states are never reused

The trainer computes the current policy probability and forms an importance ratio comparing it to the behavior policy probability. Proposition 3.1 formalizes the payoff: moving the prompt-response split leaves the conditional distribution unchanged for a fixed token history. This is mathematical equivalence — different kernels and numerical precision can still cause numerical discrepancies.

💡 Tip for ML Engineers: If your architecture has any recurrent component, old states become stale after parameter updates. Reconstruct states from the sequence start under current parameters. Record behavior log-probabilities including all sampling transformations — metadata alone cannot restore missing support from truncated sampling.

Training and Serving

Pretraining

Pretraining is full-sequence next-token prediction with full backpropagation through time (BPTT). Independent documents reset:

• Recurrent outputs
• Decoder SWA caches
• Encoder caches
• Positions

Supervised Fine-Tuning (SFT)

Loss is masked to assistant targets only, but state updates are never masked. Gradients from assistant losses backpropagate through user and tool tokens. The state is not reset at an assistant boundary.

💡 Tip — The SFT Insight: The model learns how to "think about" the user's message, not just how to respond to it. Loss masking removes the loss term, not the computation or the gradient path.

Truncated BPTT

Appendix B shows why partial detaching is risky. The state-to-state Jacobian has cross terms through the decoder key-value cache. The recurrent state has two components:

1. Final decoder output
2. Decoder SWA cache

Detaching only the final decoder output leaves gradient paths through the SWA cache. Therefore, any truncated-BPTT scheme must explicitly identify every detached tensor.

💡 Tip: Detaching the recurrent hidden state alone is NOT sufficient to cut temporal gradient dependencies. The decoder SWA cache also carries gradient information. A complete detach requires stop-gradient on both components of the recurrent state.

Multi-Turn Serving

An exact prefix snapshot includes:

• Encoder cache
• Encoder memory
• Complete decoder state
• Position metadata
• Window convention
• Model version

A fixed-weight snapshot can be reused because the state is independent of the serving split. Weight updates invalidate old states. Editing a prefix forces recomputation from an earlier checkpoint. External tokens in multi-turn RL update the state but get no importance-ratio factors.

How It Relates to Prior Work

Encoder-derived memory follows:

• YOCO (2024) — caches KV once for a cross-decoder
• DeepSeek-V4.1-Flash (2026) — projects decoder global KV from final encoder states

RLT keeps the encoder-derived memory but drops prompt-wide decoder skipping — every prompt token gets its full decoder update.

Temporal feedback builds on:

• Feedback Transformer (2020) — carries information from processed tokens into future computation
• Recurrent Transformer (2026) — forms each layer's persistent KV from that layer's output

RLT instead feeds the previous final decoder output into the next decoder input and runs recurrence over the prompt too.

Depth-wise reuse connects to:

• Universal Transformers (2018) — share computation across depth
• Recurrent-depth latent reasoning (2025) — iterates a recurrent block to scale computation

The RL replay argument extends Zhang's prefill-decode kernel mismatch note (2026).

Comparison: RLT vs Prior Architectures

Key Takeaways

1. RLT carries the full decoder state — final output plus layerwise SWA cache — across every prompt and response token with no boundary reset

2. Reference configuration: 48 tied encoder + 48 tied decoder layers = 96 logical blocks per token. State path has structural depth of 48t blocks after t tokens

3. Hardware opportunities: Encoder parallelism and batching across sequences. No parallel scan or reduced-prefill speedup is claimed

4. RL replay rebuilds all states under current parameters while keeping recorded behavior log-probabilities as ratio denominators

5. No measured results: Reasoning quality, efficiency, and RL scaling remain open validation targets

Practical Tips

Tip #1: Chain Your Prompts — Don't Dump Everything at Once

More rounds mean more reasoning passes through the model's layer stack on accumulated context. Instead of one massive prompt, build context iteratively through back-and-forth conversation.

Tip #2: Recognize "State Reset" Failures in Production

When AI contradicts earlier context or forgets constraints, don't just rephrase. Instead:

• Restate key constraints in your current message, or
• Start a fresh conversation and rebuild context deliberately

Tip #3: Audit Your RL State Handling

If building RL training pipelines, check:

• Are training states computed under current parameters?
• Are you reusing stale rollout states?
• Are behavior log-probabilities recorded correctly with all sampling transforms?

Reconstruct states from the sequence start under current parameters — never reuse old states after parameter updates.

References

1. RLT — Zhang, Y. (2026) — GitHub
2. YOCO — Sun et al. (2024) — arXiv:2405.05254
3. Feedback Transformer — Fan et al. (2020) — arXiv:2002.09402
4. Recurrent Transformer — Oncescu et al. (2026) — arXiv:2604.21215
5. Universal Transformers — Dehghani et al. (2018) — arXiv:1807.03819
6. Recurrent Depth — Geiping et al. (2025) — arXiv:2502.05171
7. Prefill-Decode Mismatch — Zhang et al. (2026) — GitHub PDF
8. DeepSeek-V4.1-Flash — DeepSeek-AI (2026) — HuggingFace

Check out the Technical Report, GitHub repository, and Project Page. All credit goes to the researcher of this project.