meta-pytorch / meta-pytorch/data
FileExistsError when using `on_disk_cache` and multiple workers
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
When multiple workers try to create a folder structure at the same time it fails and raises FileExistsError.
Expected behavior: create folder structure and continue
torchdata.datapipes.iter.IterableWrapper(urls)\
.on_disk_cache(filepath_fn=filepath_fn)\
.open_files_by_fsspec(mode='rb')\
.end_caching(filepath_fn=filepath_fn)
dl = DataLoader(datapipe, batch_size=16, num_workers=96, persistent_workers=True)
Versions
torchdata==0.6.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the on_disk_cache entry point and reproduce the issue with the provided DataLoader configuration using multiple workers. Trace the folder-creation path and verify that concurrent workers can create the folder structure and continue without FileExistsError; the supplied pipeline should complete successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100