State of Routing in Model Serving

Netflix's Switchboard processes 1 million requests per second, providing centralized ML abstraction for clients.

• •Model serving encompasses preprocessing, computation, and inference as unified workflows
• •"Objective" abstractions guide routing and decouple clients from models
• •Context-aware routing considers user device, locale, and A/B test status
• •Dynamic traffic splitting enables canary deployments and gradual rollouts
• •Enables rapid ML iteration while shielding clients from changes

This summary was automatically generated by AI based on the original article and may not be fully accurate.