How we optimized Dash's relevance judge with DSPy

Dropbox engineering shares how they used DSPy to optimize their LLM-based relevance judge for Dash, achieving significant cost and quality improvements.

• •The relevance judge scores query-document pairs on a 1–5 scale and is used across ranking, training data generation, and offline evaluation pipelines.
• •Human-annotated examples and Normalized Mean Squared Error (NMSE) were used as the ground truth metric to measure judge quality.
• •DSPy's GEPA optimizer was applied to adapt a prompt tuned for OpenAI's o3 to a cheaper open-weight model (gpt-oss-120b) by iteratively generating structured feedback on model-vs-human disagreements.
• •The optimized prompt reduced NMSE by 45% (8.83 → 4.86) and cut model adaptation time from 1–2 weeks to 1–2 days.
• •