dlt-hub / dlt-hub/dlt

`normalize` fails with `Directory not empty` when the pipeline working dir is on NFS (silly-renamed `.nfs*` file inside `.restore/schemas`), and the leftover package breaks every subsequent run

Open
#4,446 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.30.0

### Describe the problem

When `pipelines_dir` is on an NFS mount, the normalize step intermittently fails while deleting the extracted package after it has been committed to `load/normalized`:

```
File "dlt/normalize/normalize.py", line 286, in spool_files
self.normalize_storage.extracted_packages.delete_package(load_id)
File "dlt/common/storages/load_package.py", line 786, in delete_package
self.storage.delete_folder(package_path, recursively=True)
File "dlt/common/storages/file_storage.py", line 106, in delete_folder
shutil.rmtree(folder_path, onerror=del_ro)
...
OSError: [Errno 39] Directory not empty: 'schemas'
dlt.pipeline.exceptions.PipelineStepFailed: Pipeline execution failed at `step=normalize` when processing package with `load_id=1789029472.7672272`
```

The directory that cannot be removed is `normalize/extracted//.restore/schemas`, and it contains a single NFS silly-renamed file (`.nfs0000000002c800480000003e`). That file disappears on its own ~15 ms later. Nothing holds it open (`fuser`/`lsof` are empty); the schema blob was read with a `with open(...)` block that had already exited.

By the time the exception is raised the package has already been renamed into `load/normalized/` and its jobs are intact, so the data itself is fine. But the half-deleted `normalize/extracted/` (now missing `schema.json`) is left behind, and every subsequent `pipeline.run()` tries to resume it and fails deterministically:

```
FileNotFoundError: [Errno 2] No such file or directory: '.../normalize/extracted/1789029472.7672272/schema.json'
```

so one transient NFS hiccup turns into a permanently broken pipeline until someone deletes the directory by hand.

### Expected behavior

Either the delete succeeds (retry on `ENOTEMPTY`), or a package whose extracted directory is partially deleted is not treated as pending on the next run.

### Steps to reproduce

The race is reproducible without dlt. On an NFSv4 mount with the `fsc` option (FS-Cache via `cachefilesd`), mimic what normalize does — write a file atomically, read it back, then `rmtree` the tree:

```python
import os, shutil
base = '/mnt/nfs/backup/rmtest'
fails = 0
for i in range(400):
d = os.path.join(base, 'pkg', '.restore', 'schemas'); os.makedirs(d, exist_ok=True)
tmp, final = os.path.join(d, 'tmp.json'), os.path.join(d, 'x.schema.json')
with open(tmp, 'w') as f: f.write('{"a":1}' * 200)
os.rename(tmp, final)
with open(final) as f: f.read()
try:
shutil.rmtree(os.path.join(base, 'pkg'))
except OSError as e:
fails += 1; print(i, e, os.listdir(d))
shutil.rmtree(os.path.join(base, 'pkg'), ignore_errors=True)
print(fails, '/ 400')
```

Output on my box: 3 / 400 failures (`Directory not empty: 'schemas'`, one `.nfs*` entry each time, gone within ~15 ms).

With dlt, any pipeline whose `pipelines_dir` is on such a mount hits this roughly once per hundred-odd runs; each run deletes one extracted package right after reading `.restore/schemas/.schema.json` (`load_storage.import_extracted_package` → `spool_files`), which is the file that gets silly-renamed.

Mount: `nfs4 (rw,relatime,vers=4.2,...,fsc,...)`, `cachefilesd` running. I have not been able to test without `fsc` (no root), so I can't say whether a plain NFS mount is affected; the mechanism (NFS silly rename after a lingering inode reference) would apply to any NFS client that delays the release past `close()`.

### Operating system

Linux

### Runtime environment

Local

### Python version

3.12

### dlt data source

_No response_

### dlt destination

Filesystem & buckets

### Other deployment details

### Operating system
Linux 6.8.0-138-generic, Python 3.12.3, destination `filesystem` (local parquet), source is a plain `@dlt.resource` generator with `dlt.sources.incremental`.

### Additional information

### Suggested fix
`FileStorage.delete_folder` could retry `shutil.rmtree` a few times with a short sleep on `OSError` with `errno.ENOTEMPTY` (the silly-renamed entry is gone within tens of milliseconds), which is what `delete_package` callers in `normalize.py` and `pipeline.py` implicitly rely on. Separately, `normalize` could treat an extracted package without `schema.json` as already-committed garbage rather than a pending package, since the commit (`commit_new_load_package`) has already happened at that point.

---
Diagnosed with Claude Fable 5.1 (Claude Code); the repro script and traceback are from a real run.

Contributor guide

Open the contributing guide

Research direction

Start with dlt/common/storages/file_storage.py at delete_folder and dlt/common/storages/load_package.py at delete_package, then trace dlt/normalize/normalize.py at spool_files and the extracted-package handling around load_storage.import_extracted_package and commit_new_load_package. Use the NFS reproduction to exercise cleanup; done means transient ENOTEMPTY cleanup no longer leaves a package that breaks the next pipeline.run(), or the partial package is safely excluded from pending work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.