apache / apache/druid

Temporary working folders are left behind on Middle Managers after tasks complete

Open
#12,332 8 comments 0 reactions 0 assignees View on GitHub
Area - Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version
Apache Druid 0.22.1

### Description

This problem was originally reported here: https://www.druidforum.org/t/temp-folder-size-was-increasing-due-to-that-peons-processing-taking-more-time-how-to-clear-temp-folder-automatically/7139

I was able to reproduce it by running on a small minikube deployment by running the vanilla wikipedia index_parallel ingestion a few times, each with a different target datasource name and confirmed that after the jobs completed the temporary folders for the tasks are not being removed, after 3 runs, the ~/var/tmp folder still contained the three empty folders:

```
~/var/tmp $ ls -l
total 12
drwx------ 2 druid druid 4096 Mar 14 23:39 druid-realtime-persist1040350100896362009
drwx------ 2 druid druid 4096 Mar 14 23:32 druid-realtime-persist668375622911252079
drwx------ 2 druid druid 4096 Mar 14 23:34 druid-realtime-persist944793843865837077
~/var/tmp $ ls -l druid-realtime-persist944793843865837077
total 0
~/var/tmp $ ls -l druid-realtime-persist668375622911252079
total 0
~/var/tmp $ ls -l druid-realtime-persist1040350100896362009
total 0
```

The original report on Druid Forum spoke of thousands of such folders left behind.

Contributor guide

Open the contributing guide

Research direction

Start by tracing temporary-folder creation and cleanup through the Middle Manager task lifecycle, then reproduce the issue with repeated vanilla Wikipedia index_parallel jobs on a small deployment. Done means completed tasks no longer leave empty task folders under ~/var/tmp.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.