huggingface / huggingface/datatrove
Question: could `url_filterblacklistsv0_3_0.tar.gz` be hosted externally?
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Hi! First of all, thanks for this library — really impressive work.
While reading through the code I noticed that `src/datatrove/assets/url_filterblacklistsv0_3_0.tar.gz` (~17 MB) lives in the source tree. I might be missing context about why it's bundled this way, but I was wondering if it could eventually be hosted on HuggingFace Hub and downloaded on first use — since `url_filter.py` already uses `huggingface_hub.cached_assets_path` for the extracted files, it feels like a natural fit.
No rush at all, just a thought. If it's intentional or there's a reason to keep it local I completely understand!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.