Bridging the Gap: Diagnosing Online–Offline Discrepancy in Pinterest’s L1 Conversion Models

Pinterest investigates the online–offline discrepancy in L1 CVR models in their ads funnel.

• •New CVR models showed 20–45% LogMAE reduction offline but neutral or negative online A/B results.
• •Offline bugs, exposure bias, and serving health were ruled out as root causes.
• •Feature gap: models trained on rich signals (targeting flags, conversion counts, image embeddings) absent from L1 serving.
• •Embedding version skew in two-tower models caused query and Pin checkpoint misalignment at serving.