EMO: Pretraining mixture of experts for emergent modularity

EMO is a mixture-of-experts model trained to develop modular expert groups that can be selectively used for specific tasks.

• •Only 12.5% of total experts needed for task-specific performance with minimal degradation
• •Document boundaries used as training signal to encourage experts to specialize by semantic domain
• •Global load balancing resolves conflicts between modularity and expert utilization objectives
• •Expert clusters correspond to meaningful domains like health, politics, and music rather than syntactic features
• •Maintains strong general-purpose performance when all 128 experts are used together

This summary was automatically generated by AI based on the original article and may not be fully accurate.