Develop a tool to validate kafka sink output across new arch and old arch

Is your feature request related to a problem?

Validating the correctness of a Kafka sink is challenging without understanding the downstream business logic, which often requires significant effort from the customer and is usually not feasible. Therefore, we need a way to verify data consistency between the old and new architectures before upgrading, without relying on domain-specific knowledge.

Describe the feature you'd like

The proposed feature is a data comparison tool that automatically consumes and aligns output streams from both the old and new Kafka sink architectures. It would compare messages and generate a clear report highlighting any discrepancies. This would provide confidence in data correctness before switching production traffic, independent of business logic.

Describe alternatives you've considered

No response

Teachability, Documentation, Adoption, Migration Strategy

No response