Add support for slowly changing mapping tables
Aperta
Conformance
feature
priority: medium
under discussion
- Lingua principale
- Scala
- Stelle
- 33
- Fork
- 16
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Background
Streaming conformance currently uses static mapping tables read from HDFS.
If a job is running long enough the mapping tables become outdated.
## Feature
Long-running streaming conformance jobs need a way to make in-memory mapping tables up to date.
Not sure yet how to pick up the changes to mapping tables.
- [ ] Periodically check the HDFS folder with mapping tables for updates
- [ ] Subscribe to a Kafka topic with changes to mapping tables
Which way to go depends on how MDM and MDR are done.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.