Databricks announces General Availability of Real-Time Mode (RTM) for Apache Spark Structured Streaming, enabling millisecond-level latency without leaving the Spark ecosystem.
- •RTM eliminates the need for separate streaming engines like Apache Flink by bringing sub-millisecond latency directly to existing Spark APIs
- •Three architectural innovations power RTM: continuous data flow (event-by-event processing), pipeline scheduling (non-blocking stage execution), and streaming shuffle (bypassing disk-based shuffle bottlenecks)
- •Benchmarks show RTM outperforms Apache Flink by up to 92% on feature computation workloads representative of fraud detection and personalization use cases
- •Coinbase achieved 80%+ reduction in end-to-end latencies with sub-100ms P99s; MakeMyTrip delivered sub-50ms P50 latencies with a 7% uplift in click-through rates
- •New GA features include OSS support in Apache Spark 4.1, async state checkpointing, initial state load in transformWithState, and enhanced Python
This summary was automatically generated by AI based on the original article and may not be fully accurate.
Databricks
Google Cloud
AWS