huggingface / huggingface/datatrove
[FEATURE] CC Filter
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
I was looking at dolma, and they have a nice filter to filter out CreativeCommons data only. It might be worthwhile to add something similar to datatrove, too.
https://github.com/allenai/dolma/blob/64886d9db15bd99acea9e28740ae20a510875dfb/python/dolma/taggers/licenses.py#L19
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.