Five techniques to reach the efficient frontier of LLM inference

This article explains the concept of an efficient frontier for LLM inference and five practical techniques to reach it.

• •LLM inference has two phases: prefill (compute-bound) and decode (memory-bandwidth-bound), creating an inherent latency/throughput tradeoff
• •Semantic routing directs simple queries to smaller models and complex ones to frontier models, maximizing throughput without sacrificing quality
• •Prefill/decode disaggregation runs each phase on specialized hardware clusters to push both toward their theoretical limits independently
• •Quantization (AWQ, GPTQ) reduces weights from FP16 to INT4, enabling up to 4x faster memory reads during the decode phase
• •Context-aware L7 routing directs requests to pods already holding matching KV cache prefixes, cutting TTFT by up to 85%