DruidInputSource does not close segments properly
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
There are two issues:
- `DruidSegmentReader` calls `indexIO.loadIndex` but does not ever close the result. So, the file remains memory-mapped even after we no longer need it.
- `DruidSegmentInputEntity` uses `segmentFile.delete()` to delete segment files, but this doesn't work. Segments are
directories, so File.delete won't work on them. Even if it did work, it wouldn't reclaim the disk space, since the file is still in use due to the lack of unmapping.
The latter one leads to a bunch of log messages involving `Could not clean temporary segment file`.
See also PR #10986, where I tried a simple fix but was foiled due to objects escaping in the `intermediateRowIterator`.
Note that MSQ does not have this problem. `TaskDataSegmentProvider` closes segments and deletes the local disk files properly. It doesn't run into the "escaping objects" problem, because objects are copied to frames if needed, rather than being retained in heap, after processing a segment.
Contributor guide
Research direction
Start with DruidSegmentReader and DruidSegmentInputEntity, then compare their segment lifecycle with TaskDataSegmentProvider. Review PR #10986 and the intermediateRowIterator escaping concern before determining how loaded indexes and segment directories can be released safely. Done means temporary segment files are cleaned up without retaining objects needed after processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100