Add support for slowly changing mapping tables
Abierto
Conformance
feature
priority: medium
under discussion
- Lenguaje dominante
- Scala
- Estrellas
- 33
- Forks
- 16
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
## Background
Streaming conformance currently uses static mapping tables read from HDFS.
If a job is running long enough the mapping tables become outdated.
## Feature
Long-running streaming conformance jobs need a way to make in-memory mapping tables up to date.
Not sure yet how to pick up the changes to mapping tables.
- [ ] Periodically check the HDFS folder with mapping tables for updates
- [ ] Subscribe to a Kafka topic with changes to mapping tables
Which way to go depends on how MDM and MDR are done.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.