Databricks presents Spark Declarative Pipelines and Enzyme for incremental data processing at SIGMOD 2026.
- •Enzyme engine incrementally maintains Materialized Views, hiding complexity from data engineers
- •Data engineers can use either Materialized Views or streaming APIs for incremental processing in production workloads
- •Enzyme supports complex patterns including joins, window functions, aggregations, non-deterministic functions, and AI functions
- •Multi-language support for both SQL and Python specifications for Materialized View definitions
- •Performance optimizations including partition-level updates and selective caching reduce processing overhead
This summary was automatically generated by AI based on the original article and may not be fully accurate.
Databricks
Google Cloud
Cloudflare