huggingface / huggingface/datatrove

Question: could `url_filterblacklistsv0_3_0.tar.gz` be hosted externally?

Open
#477 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

Hi! First of all, thanks for this library — really impressive work.

While reading through the code I noticed that `src/datatrove/assets/url_filterblacklistsv0_3_0.tar.gz` (~17 MB) lives in the source tree. I might be missing context about why it's bundled this way, but I was wondering if it could eventually be hosted on HuggingFace Hub and downloaded on first use — since `url_filter.py` already uses `huggingface_hub.cached_assets_path` for the extracted files, it feels like a natural fit.

No rush at all, just a thought. If it's intentional or there's a reason to keep it local I completely understand!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.