mlcommons / mlcommons/storage

Intermittent errors between epochs

Open
#499 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug DLIO or mlpstorage Future
Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

We see these errors intermittently between epochs.  Not every test.  And, not every epoch.  In this case, it occurred once out of 8 epochs. The errors do not prevent the test from completion. We first noticed it after re-cloning to 3.0.15.

[OUTPUT] 2026-06-20T15:21:32.972188 Ending epoch 4 - 32641 steps completed in 1594.25 s
[OUTPUT] 2026-06-20T15:21:33.499306 Reshard for epoch 5: 783387 train files, 0 eval files redistributed via alltoall in 0.53s
Traceback (most recent call last):
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
finalizer()
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in call
res = self._callback(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
rmtree(tempdir, onerror=onerror)
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
_rmtree_safe_fd(stack, onexc)
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
onexc(func, path, err)
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
return onerror(func, path, exc_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 662, in _rmtree_safe_fd
os.rmdir(name, dir_fd=dirfd)
OSError: [Errno 39] Directory not empty: '/tmp/pymp-9b8nyjch'
[OUTPUT] 2026-06-20T15:21:35.302933 Worker pre-warm complete for epoch 5 (8 workers spawned)
[OUTPUT] 2026-06-20T15:21:35.307140 Starting epoch 5: 32641 steps expected

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names Python multiprocessing finalizer cleanup, shutil.rmtree, and the reshard/worker pre-warm transition, but no repository file or test. Reproduce the intermittent error across repeated epoch transitions and trace temporary-directory cleanup around resharding. Done means the reported Directory not empty traceback no longer appears during completed runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.