huggingface / huggingface/faceberg

Implementation of `dataset_builder_safe` is not thread safe

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
31
Forks
2
Avg merge
23m
Merged PRs (30d)
1

Description

In the `discovery.py` module, the [dataset_builder_safe](https://github.com/huggingface/faceberg/blob/2346ee32ba68a7f906e1244d4b9056c404f35722/faceberg/discover.py#L18) uses `os.chdir()` to change the current working directory for the entire Python process. Other functions such as `write_manifest()` apply `ThreadPoolExecutor` for parallel I/O work. If two threads are working at the same time, one thread can change the working directory while another thread is trying to access a relative path. This leads to `FileNotFoundError` errors which are hard to diagnose. WDYT?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the dataset_builder_safe implementation in the referenced discovery.py/discover.py module, then inspect write_manifest and its ThreadPoolExecutor use. Trace how os.chdir() and relative paths interact during concurrent I/O; done means parallel work no longer encounters working-directory-dependent FileNotFoundError failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.