huggingface / huggingface/datatrove
Docs idea: add a small dataset-audit example using filters, exclusion_writer, and summary stats
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Hi DataTrove maintainers,
I was reading the README and examples while comparing public LLM data-processing tools for dataset-quality workflows. DataTrove already documents the core pieces that are useful for an audit-style pipeline:
- filters and rejected-sample capture via `exclusion_writer`
- output metadata fields on `Document`
- summary statistics and `StatsMerger`
- larger examples such as `examples/fineweb.py` and `examples/summary_stats.py`
One small documentation addition that may help practitioners is a compact “dataset audit” example showing how to combine these pieces before publishing or reusing a corpus. The example could stay lightweight and avoid introducing new code paths, for example:
1. read a small public JSONL sample,
2. apply one or two quality filters with `exclusion_writer`,
3. preserve simple provenance metadata such as source or dump,
4. collect summary stats before/after filtering,
5. write a short checklist of artifacts to review: kept docs, rejected docs, stats, logs, and completion markers.
This would not need to prescribe a universal quality bar. It would just show how DataTrove users can make filtering decisions more reviewable and reproducible.
If this fits the project direction, I would be happy to draft a small docs PR scoped to the existing README/examples style. If maintainers prefer a different location or framing, I can adapt.
Thanks for maintaining the project.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.