Monitoring reliably at scale

This post explains how Airbnb eliminated circular dependencies in its observability stack to ensure reliable monitoring at scale.

• •Dedicated Kubernetes clusters for observability workloads, isolated from product applications but managed by the Cloud team, reducing shared failure domains
• •Custom Layer 7 network ingress based on Envoy to replace Istio, enabling strict prioritization and isolation of observability traffic from business traffic
• •Header-based tenant routing system to map over 1,000 services for predictable load distribution
• •Meta-monitoring layer with isolated Prometheus instances across different availability zones to monitor the observability stack itself
• •Dead Man's Switch mechanism as a signal-based failsafe to detect meta-monitoring failures without infinite regression