fsspec / fsspec/filesystem_spec
makedirs is expensive on NFS
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I was working on an NFS today with Zarr and was surprised to see that storing data was very slow. Looking at a profile I saw that the culprit was a call to os.makedirs (sorry, I didn't get the exact line, and this was on someone else's computer).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the os.makedirs call mentioned in the report and reproduce the profiling result while storing Zarr data on NFS. Identify the responsible filesystem entry point, then validate the behavior with a comparable NFS performance check; done means the reported slowdown is understood and the unnecessary cost is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100