Too many open files error in indexTask
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
`AppenderatorImpl` holds all generated segment files until `abandonSegments()` is called. `IndexTask` works based on `AppenderatorImpl` and calls `abandonSegments()` once it pushes the segments to deep storage. This can cause an issue of too many open files if the indexTask is running with the perfect rollup mode. In the perfect rollup mode, the indexTask can't push segments in the middle of indexing. Instead, they should hold all segments until the very end of the indexing.
The way how `AppenderatorImpl` works makes sense for stream ingestion because the ingested data should be available for query immediately. However, batch tasks don't have to hold them.
Contributor guide
Research direction
Start by tracing how AppenderatorImpl retains generated segment files, how IndexTask invokes abandonSegments(), and how perfect rollup changes when segments are pushed to deep storage. Compare the batch and stream-ingestion lifecycles, then define completion as avoiding excessive open files for batch perfect-rollup tasks without removing immediate availability for stream ingestion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100