BootstrapOperator will load all records while bootstrap parallelism increased, ListState and SPLIT_DISTRIBUTE is not suitable for the operator?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Expected behavior**
Load records was slow while I increased the parallelism of the bootstrap operator, Caused by new subtask could not get the lastInstantTime because the ListState is SPLIT_DISTRIBUTE redistribution mode.
When I changed the mode and then the subtask failed caused by
Is there any suggestions for resolving the large table failover easily? This load action may take long time and the flink task is not initialled for long time.
**Environment Description**
* Hudi version :0.11
* Flink version: 1.15
**Additional context**
**Stacktrace**
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the BootstrapOperator entry point and trace how its ListState uses SPLIT_DISTRIBUTE during bootstrap and failover on Flink 1.15. Reproduce the increased-parallelism case and verify that recovery avoids loading all records and does not produce the incompatible state-assignment-mode error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100