Run real-time and async inference on the same infrastructure with GKE Inference Gateway

This post introduces GKE Inference Gateway, a unified platform on Google Kubernetes Engine for running both real-time and async AI inference workloads on shared GPU/TPU infrastructure.

• •Real-time inference uses latency-aware scheduling based on metrics like KV cache utilization to minimize time-to-first-token under heavy load.
• •Async inference is handled by an Async Processor Agent that integrates with Cloud Pub/Sub, routing batch requests as 'sheddable' traffic during idle accelerator cycles.
• •Real-time traffic always takes strict priority at the gateway level; async tasks fill unused compute capacity between real-time spikes.
• •Initial testing showed that without the Async Processor, unmanaged multiplexing caused 99% message drop; with it, 100% of latency-tolerant requests were served.