apache / apache/pinot

Local temp files are not always cleaned up

Open
#11,573 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Over time the temporary .tar.gz files accumulate on servers' local disk, requiring manual cleanup. This could be improved with a periodic task to ensure leftover files are removed.

Sample accumulation over a couple months:
```
[docker@<...>/dataDir/_REALTIME #]l
consumers/ __29__760__20230813T2316Z.tar.gz __5__1902__20230820T0323Z.tar.gz
__11__1180__20230816T0319Z.tar.gz __31__1891__20230820T0321Z.tar.gz __53__1105__20230723T0431Z.tar.gz
__1__11983__20230715T0116Z.tar.gz __31__639__20230813T0705Z.tar.gz __53__2025__20230820T1927Z.tar.gz
__1__12014__20230715T0839Z.tar.gz __33__1111__20230723T0150Z.tar.gz __53__906__20230720T1949Z.tar.gz
__1__12730__20230722T0925Z.tar.gz __33__1263__20230724T1657Z.tar.gz __57__1014__20230815T0707Z.tar.gz
__1__1625__20230818T1428Z.tar.gz __33__1557__20230818T0600Z.tar.gz __57__1311__20230816T2142Z.tar.gz
__13__1943__20230820T0952Z.tar.gz __33__242__20230811T0203Z.tar.gz __57__317__20230715T0121Z.tar.gz
__13__763__20230813T2304Z.tar.gz __33__995__20230815T0357Z.tar.gz __57__762__20230813T2315Z.tar.gz
__15__1867__20230819T2226Z.tar.gz __35__703__20230813T1444Z.tar.gz __5__9398__20230707T2025Z.tar.gz
__17__244__20230811T0232Z.tar.gz __35__713__20230813T1601Z.tar.gz __5__9582__20230708T0259Z/
__17__443__20230716T0333Z.tar.gz __41__1135__20230815T2218Z.tar.gz __5__9583__20230708T0301Z/
__17__480__20230716T1042Z.tar.gz __41__1565__20230818T0754Z.tar.gz __5__9584__20230708T0302Z/
__17__615__20230717T1808Z.tar.gz __41__914__20230814T1813Z.tar.gz __59__990__20230815T0358Z.tar.gz
__1__769__20230813T2313Z.tar.gz __43__1458__20230817T1614Z.tar.gz __61__1031__20230722T0821Z.tar.gz
__19__237__20230811T0200Z.tar.gz __43__1976__20230820T1304Z.tar.gz __61__108__20230712T2332Z.tar.gz
__1__9434__20230707T1411Z/ __45__122__20230712T2323Z.tar.gz __61__1881__20230820T0033Z.tar.gz
__1__9546__20230707T1919Z/ __45__1458__20230817T1618Z/ __61__359__20230811T1710Z.tar.gz
__19__659__20230813T1001Z.tar.gz __45__1458__20230817T1618Z.tar.gz __61__666__20230813T1013Z.tar.gz
__1__9840__20230708T2244Z.tar.gz __47__1141__20230815T2218Z.tar.gz __63__1903__20230820T0331Z.tar.gz
__1__9899__20230709T0025Z/ __47__1458__20230817T1613Z.tar.gz __63__2183__20230821T1552Z.tar.gz
__21__1245__20230724T1421Z.tar.gz __47__1848__20230819T2002Z.tar.gz __7__798__20230814T0324Z.tar.gz
__21__418__20230715T2322Z.tar.gz __47__242__20230811T0205Z.tar.gz __9__1654__20230818T1839Z.tar.gz
__23__565__20230812T2025Z.tar.gz __49__1244__20230724T1656Z.tar.gz __9__243__20230811T0212Z.tar.gz
__25__1568__20230818T0803Z.tar.gz __49__131__20230712T2337Z.tar.gz __9__445__20230715T2043Z.tar.gz
__25__1894__20230820T0320Z.tar.gz __49__1622__20230818T1425Z.tar.gz segment-stats.ser
__25__240__20230811T0207Z.tar.gz __5__12051__20230712T2046Z.tar.gz _tmp/
__29__157__20230713T1035Z.tar.gz __5__12076__20230713T0023Z.tar.gz tmp/
__29__1588__20230818T1133Z.tar.gz __51__700__20230813T1436Z.tar.gz

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing where the temporary .tar.gz files are created and retained in realtime table data directories. Determine the safe cleanup conditions and verify that periodic cleanup removes leftover archives without affecting active work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.