apache / apache/druid

Too many open files error in indexTask

Open
#6,286 2 comments 0 reactions 0 assignees View on GitHub
Bug stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

`AppenderatorImpl` holds all generated segment files until `abandonSegments()` is called. `IndexTask` works based on `AppenderatorImpl` and calls `abandonSegments()` once it pushes the segments to deep storage. This can cause an issue of too many open files if the indexTask is running with the perfect rollup mode. In the perfect rollup mode, the indexTask can't push segments in the middle of indexing. Instead, they should hold all segments until the very end of the indexing.

The way how `AppenderatorImpl` works makes sense for stream ingestion because the ingested data should be available for query immediately. However, batch tasks don't have to hold them.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how AppenderatorImpl retains generated segment files, how IndexTask invokes abandonSegments(), and how perfect rollup changes when segments are pushed to deep storage. Compare the batch and stream-ingestion lifecycles, then define completion as avoiding excessive open files for batch perfect-rollup tasks without removing immediate availability for stream ingestion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.