huggingface / huggingface/datatrove
Migrate word tokenizer download functions to process locked download
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
## Problem
#187 Introduced new tokenizer libraries, which will often need to download several files to work. This can however introduce a problems as the downloads are not interlocked.
## Solution
Inspect the libraries and try to pre-download the files using process-locked download
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect the word tokenizer download functions and the tokenizer libraries introduced by #187. Determine which files each library downloads, then verify that pre-downloads use the process-locked download mechanism and cover all required files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100