apache / apache/hudi

HoodieMultiTableDeltaStreamer does not ingest from all topics when using continuous mode

Open
#14,789 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:ingest from-jira priority:high status:pr-available status:triaged type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1881
- Type: Bug

---

## Comments

16/May/21 03:08;nishith29;Looking into the HoodieMultiTableDeltaStreamer is not designed to run in continuous mode as can be seen, the execution is done serially and in continuous mode, it's stuck at the first one.

 
{code:java}
for (TableExecutionContext context : tableExecutionContexts) {
try {
new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
successTables.add(Helpers.getTableWithDatabase(context));
} catch (Exception e) {
logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e);
failedTables.add(Helpers.getTableWithDatabase(context));
}
}
{code}
 

 ;;;

---

16/May/21 23:15;nishith29;To enable this feature, we might have to implement a threadpool and then let the sync() for each table happen in asynchronously without blocking.  ;;;

---

18/May/21 00:09;shivnarayan;[~nishith29]: I guess we can downgrade the severity to sev:high IMO. ;;;

---

13/Dec/21 14:15;shivnarayan;CC [~harsh1231] ;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.