Zerobus Ingest is Databricks' serverless streaming service for petabyte-scale data ingestion into Delta tables.
- •Ingested 1 petabyte of NASA NEOWISE data in 24 hours with 12 GB/s sustained throughput and zero pre-configuration.
- •Uses dynamic partitioning with stream-level ordering guarantees to enable autoscaling without manual broker provisioning.
- •Custom ZeroParser protobuf decoder achieves 1 GB/s per CPU core using single-pass parsing with zero memory allocations.
- •Implements latency-optimized write-ahead log (WAL) with async acknowledgements for Kafka-like durability guarantees.
This summary was automatically generated by AI based on the original article and may not be fully accurate.