Building a Next-Generation Key-Value Store at Airbnb

Airbnb shares how they completely rearchitected Mussel, their internal key-value store for derived data, migrating from v1 to a NewSQL-based v2 running in production for over a year.

• •V1 suffered from operational complexity (manual Chef scripts on EC2), static hash partitioning hotspots, limited consistency control, and opaque resource usage
• •V2 adopts Kubernetes-native deployment with a stateless Dispatcher service, dynamic range sharding, and p99 reads under 25ms even for 100TB+ tables
• •Writes are persisted to Kafka first for durability, with a Replayer applying them in order to the backend, enabling burst absorption and consistency guarantees
• •Migration used a blue/green strategy with shadowing, dual writes, automatic circuit breakers, and per-table reversible cutover steps to achieve zero data loss
• •A custom pipeline handled bootstrapping via Kubernetes StatefulSets with checkpointing, checksum verification, Kafka catch-up, and gradual traffic shifting