apache / apache/druid

DruidInputSource does not close segments properly

Open
#14,486 0 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Area - Compaction Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

There are two issues:

- `DruidSegmentReader` calls `indexIO.loadIndex` but does not ever close the result. So, the file remains memory-mapped even after we no longer need it.
- `DruidSegmentInputEntity` uses `segmentFile.delete()` to delete segment files, but this doesn't work. Segments are
directories, so File.delete won't work on them. Even if it did work, it wouldn't reclaim the disk space, since the file is still in use due to the lack of unmapping.

The latter one leads to a bunch of log messages involving `Could not clean temporary segment file`.

See also PR #10986, where I tried a simple fix but was foiled due to objects escaping in the `intermediateRowIterator`.

Note that MSQ does not have this problem. `TaskDataSegmentProvider` closes segments and deletes the local disk files properly. It doesn't run into the "escaping objects" problem, because objects are copied to frames if needed, rather than being retained in heap, after processing a segment.

Contributor guide

Open the contributing guide

Research direction

Start with DruidSegmentReader and DruidSegmentInputEntity, then compare their segment lifecycle with TaskDataSegmentProvider. Review PR #10986 and the intermediateRowIterator escaping concern before determining how loaded indexes and segment directories can be released safely. Done means temporary segment files are cleaned up without retaining objects needed after processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.