A New Framework for Evaluating Voice Agents (EVA)

EVA is a new end-to-end evaluation framework for conversational voice agents that jointly measures task accuracy and conversational experience.

• •EVA produces two scores: EVA-A (Accuracy) and EVA-X (Experience), evaluated via a bot-to-bot audio architecture with 5 core components
• •EVA-A measures task completion (deterministic), faithfulness (LLM-as-Judge), and speech fidelity (LALM-as-Judge) at the audio level
• •EVA-X measures conciseness, conversation progression, and turn-taking for natural spoken interaction quality
• •Released with an airline dataset of 50 scenarios and benchmarks for 20 cascade and audio-native systems
• •Key finding: a consistent Accuracy-Experience tradeoff exists — agents that excel at task completion tend to deliver worse conversational experiences