Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

This post introduces Coban, Grab's platform for real-time Kafka stream data quality monitoring using user-defined data contracts with syntactic and semantic test rules.

• •Data contracts define Kafka stream schema agreements, field-level semantic rules (string patterns, number ranges, constant values), and observability metadata for alerting.
• •LLM-based recommendations assist users in predicting semantic test rules from Kafka stream schemas and anonymized sample data, reducing manual rule definition overhead.
• •The Test Runner uses FlinkSQL as the compute engine, converting human-readable semantic rules into inverse SQL queries to identify records that violate defined contracts.
• •Problematic data events are published to a dedicated Kafka topic and sinked to AWS S3; Grab's Genchi platform sends Slack notifications with sample data links and bad record counts.