`normalize` fails with `Directory not empty` when the pipeline working dir is on NFS (silly-renamed `.nfs*` file inside `.restore/schemas`), and the leftover package breaks every subsequent run
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
1.30.0
### Describe the problem
When `pipelines_dir` is on an NFS mount, the normalize step intermittently fails while deleting the extracted package after it has been committed to `load/normalized`:
```
File "dlt/normalize/normalize.py", line 286, in spool_files
self.normalize_storage.extracted_packages.delete_package(load_id)
File "dlt/common/storages/load_package.py", line 786, in delete_package
self.storage.delete_folder(package_path, recursively=True)
File "dlt/common/storages/file_storage.py", line 106, in delete_folder
shutil.rmtree(folder_path, onerror=del_ro)
...
OSError: [Errno 39] Directory not empty: 'schemas'
dlt.pipeline.exceptions.PipelineStepFailed: Pipeline execution failed at `step=normalize` when processing package with `load_id=1789029472.7672272`
```
The directory that cannot be removed is `normalize/extracted//.restore/schemas`, and it contains a single NFS silly-renamed file (`.nfs0000000002c800480000003e`). That file disappears on its own ~15 ms later. Nothing holds it open (`fuser`/`lsof` are empty); the schema blob was read with a `with open(...)` block that had already exited.
By the time the exception is raised the package has already been renamed into `load/normalized/` and its jobs are intact, so the data itself is fine. But the half-deleted `normalize/extracted/` (now missing `schema.json`) is left behind, and every subsequent `pipeline.run()` tries to resume it and fails deterministically:
```
FileNotFoundError: [Errno 2] No such file or directory: '.../normalize/extracted/1789029472.7672272/schema.json'
```
so one transient NFS hiccup turns into a permanently broken pipeline until someone deletes the directory by hand.
### Expected behavior
Either the delete succeeds (retry on `ENOTEMPTY`), or a package whose extracted directory is partially deleted is not treated as pending on the next run.
### Steps to reproduce
The race is reproducible without dlt. On an NFSv4 mount with the `fsc` option (FS-Cache via `cachefilesd`), mimic what normalize does — write a file atomically, read it back, then `rmtree` the tree:
```python
import os, shutil
base = '/mnt/nfs/backup/rmtest'
fails = 0
for i in range(400):
d = os.path.join(base, 'pkg', '.restore', 'schemas'); os.makedirs(d, exist_ok=True)
tmp, final = os.path.join(d, 'tmp.json'), os.path.join(d, 'x.schema.json')
with open(tmp, 'w') as f: f.write('{"a":1}' * 200)
os.rename(tmp, final)
with open(final) as f: f.read()
try:
shutil.rmtree(os.path.join(base, 'pkg'))
except OSError as e:
fails += 1; print(i, e, os.listdir(d))
shutil.rmtree(os.path.join(base, 'pkg'), ignore_errors=True)
print(fails, '/ 400')
```
Output on my box: 3 / 400 failures (`Directory not empty: 'schemas'`, one `.nfs*` entry each time, gone within ~15 ms).
With dlt, any pipeline whose `pipelines_dir` is on such a mount hits this roughly once per hundred-odd runs; each run deletes one extracted package right after reading `.restore/schemas/.schema.json` (`load_storage.import_extracted_package` → `spool_files`), which is the file that gets silly-renamed.
Mount: `nfs4 (rw,relatime,vers=4.2,...,fsc,...)`, `cachefilesd` running. I have not been able to test without `fsc` (no root), so I can't say whether a plain NFS mount is affected; the mechanism (NFS silly rename after a lingering inode reference) would apply to any NFS client that delays the release past `close()`.
### Operating system
Linux
### Runtime environment
Local
### Python version
3.12
### dlt data source
_No response_
### dlt destination
Filesystem & buckets
### Other deployment details
### Operating system
Linux 6.8.0-138-generic, Python 3.12.3, destination `filesystem` (local parquet), source is a plain `@dlt.resource` generator with `dlt.sources.incremental`.
### Additional information
### Suggested fix
`FileStorage.delete_folder` could retry `shutil.rmtree` a few times with a short sleep on `OSError` with `errno.ENOTEMPTY` (the silly-renamed entry is gone within tens of milliseconds), which is what `delete_package` callers in `normalize.py` and `pipeline.py` implicitly rely on. Separately, `normalize` could treat an extracted package without `schema.json` as already-committed garbage rather than a pending package, since the commit (`commit_new_load_package`) has already happened at that point.
---
Diagnosed with Claude Fable 5.1 (Claude Code); the repro script and traceback are from a real run.
Contributor guide
Research direction
Start with dlt/common/storages/file_storage.py at delete_folder and dlt/common/storages/load_package.py at delete_package, then trace dlt/normalize/normalize.py at spool_files and the extracted-package handling around load_storage.import_extracted_package and commit_new_load_package. Use the NFS reproduction to exercise cleanup; done means transient ENOTEMPTY cleanup no longer leaves a package that breaks the next pipeline.run(), or the partial package is safely excluded from pending work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100