Why We Use Separate Tech Stacks for Personalization and Experimentation

Spotify explains why they maintain separate tech stacks for personalization and experimentation rather than combining them into one system.

• •Personalization systems require ML infrastructure (boosting, neural networks, LLMs, contextual bandits) with low-latency feature access and real-time model serving that experimentation tools aren't designed to provide
• •Contextual bandits are treated as product features (personalization systems) rather than experimental methods, and must themselves be evaluated via A/B tests
• •Mixing ML and experimentation concerns creates hidden technical debt and complex dependencies between tool instances
• •Non-contextual multi-armed bandits are not used at Spotify because they optimize a single metric and cannot easily handle multi-objective trade-offs across 300+ teams
• •Their approach: ML platform handles recommendation serving, while Confidence (their experimentation platform) evaluates those systems in parallel with thousands of other experiments