apache / apache/paimon

[core][bug]Default IOManagerImpl does not clean spill folders in tempDirs when write-buffer-spillable=true

Open
#7,664 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

After paimon is configured with write-buffer-spillable=true, for write-buffer-spill.max-disk-size, the default implementation is IOManagerImpl. During the process of writing to the paimon table, is the spill folder generated under tempDirs not cleaned up

### Compute Engine

JavaAPI

### Minimal reproduce step

1、create writer:
IOManager ioManager = IOManager.create(splitPaths(tmpDirs));
return (StreamTableWrite) writeBuilder.newWrite().withIOManager(ioManager);
2、writer.close
3、look tmpDirs

### What doesn't meet your expectations?

tmpDirs exists Expired and useless directories

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at IOManager.create(splitPaths(tmpDirs)) and the default IOManagerImpl path used when write-buffer-spillable=true; trace what happens during writer.close() and spill-folder cleanup. Reproduce the listed Java API steps and verify that expired, unused directories are removed from tmpDirs after close.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.