Ai-Architecture

Mixture of Agents: How Layering Open-Source LLMs Beat GPT-4 Omni

• whackur
• Ai
• June 30, 2026

Instead of scaling a single model up, you can stack multiple models in layers and have each one refine the previous layer’s output. Together AI’s research team formalized that approach in June 2024 as Mixture of Agents (MoA) in arXiv:2406.04692. Using only open-source models, their MoA configuration scored 65.1% on AlpacaEval 2.0, versus 57.5% for GPT-4 Omni.