huggingface / huggingface/faceberg
Implementation of `dataset_builder_safe` is not thread safe
- Dominant language
- Python
- Stars
- 31
- Forks
- 2
- Avg merge
- 23m
- Merged PRs (30d)
- 1
Description
In the `discovery.py` module, the [dataset_builder_safe](https://github.com/huggingface/faceberg/blob/2346ee32ba68a7f906e1244d4b9056c404f35722/faceberg/discover.py#L18) uses `os.chdir()` to change the current working directory for the entire Python process. Other functions such as `write_manifest()` apply `ThreadPoolExecutor` for parallel I/O work. If two threads are working at the same time, one thread can change the working directory while another thread is trying to access a relative path. This leads to `FileNotFoundError` errors which are hard to diagnose. WDYT?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the dataset_builder_safe implementation in the referenced discovery.py/discover.py module, then inspect write_manifest and its ThreadPoolExecutor use. Trace how os.chdir() and relative paths interact during concurrent I/O; done means parallel work no longer encounters working-directory-dependent FileNotFoundError failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100