huggingface / huggingface/datatrove

Docs idea: add a small dataset-audit example using filters, exclusion_writer, and summary stats

Open
#485 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

Hi DataTrove maintainers,

I was reading the README and examples while comparing public LLM data-processing tools for dataset-quality workflows. DataTrove already documents the core pieces that are useful for an audit-style pipeline:

- filters and rejected-sample capture via `exclusion_writer`
- output metadata fields on `Document`
- summary statistics and `StatsMerger`
- larger examples such as `examples/fineweb.py` and `examples/summary_stats.py`

One small documentation addition that may help practitioners is a compact “dataset audit” example showing how to combine these pieces before publishing or reusing a corpus. The example could stay lightweight and avoid introducing new code paths, for example:

1. read a small public JSONL sample,
2. apply one or two quality filters with `exclusion_writer`,
3. preserve simple provenance metadata such as source or dump,
4. collect summary stats before/after filtering,
5. write a short checklist of artifacts to review: kept docs, rejected docs, stats, logs, and completion markers.

This would not need to prescribe a universal quality bar. It would just show how DataTrove users can make filtering decisions more reviewable and reproducible.

If this fits the project direction, I would be happy to draft a small docs PR scoped to the existing README/examples style. If maintainers prefer a different location or framing, I can adapt.

Thanks for maintaining the project.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.