AI evals are becoming the new compute bottleneck

AI evaluation has become a critical cost bottleneck that determines who can conduct evaluations, with the Holistic Agent Leaderboard spending $40,000 for 21,730 agent rollouts and individual GAIA runs costing $2,829.

• •Static LLM benchmarks achieved 100× to 200× cost reductions through compression techniques like Flash-HELM and tinyBenchmarks while maintaining ranking accuracy.
• •Agent evaluation is significantly more expensive and noisy, with scaffold choices creating up to 10× cost variations on identical tasks.
• •Training-in-the-loop benchmarks like The Well require massive computational resources, with evaluation costs sometimes exceeding training costs by two orders of magnitude.
• •Emerging benchmarks like PaperBench and ResearchGym demand substantial compute budgets, making them inaccessible to many research groups.