HoodieMultiTableDeltaStreamer does not ingest from all topics when using continuous mode
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1881
- Type: Bug
---
## Comments
16/May/21 03:08;nishith29;Looking into the HoodieMultiTableDeltaStreamer is not designed to run in continuous mode as can be seen, the execution is done serially and in continuous mode, it's stuck at the first one.
{code:java}
for (TableExecutionContext context : tableExecutionContexts) {
try {
new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
successTables.add(Helpers.getTableWithDatabase(context));
} catch (Exception e) {
logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e);
failedTables.add(Helpers.getTableWithDatabase(context));
}
}
{code}
;;;
---
16/May/21 23:15;nishith29;To enable this feature, we might have to implement a threadpool and then let the sync() for each table happen in asynchronously without blocking. ;;;
---
18/May/21 00:09;shivnarayan;[~nishith29]: I guess we can downgrade the severity to sev:high IMO. ;;;
---
13/Dec/21 14:15;shivnarayan;CC [~harsh1231] ;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.