huggingface / huggingface/datatrove
Fastwarc reader
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Can we add a new warc reader using the [fastwarc](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html)?
It is said to be much more [efficient](https://arxiv.org/abs/2112.03103) than warcio
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.