huggingface/datatrove

View on GitHub

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

Stars: 3.3k
Forks: 302
Open beginner issues: 0
Indexed issues: 83
Avg merge: 2h 18m
Merged PRs (30d): 2

Dominant language: Python
License: Apache-2.0
Last GitHub push: Aug 13, 2026
Latest indexed: Sep 17, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Open All

83 open issues indexed

Supporting Apache Beam Open

enhancement

huggingface/datatrove#58 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Implementation of line-wise corrections Open

enhancement

huggingface/datatrove#69 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Spark support Open

enhancement

huggingface/datatrove#70 · 3 comments · 2 reactions · 0 assignees · Indexed Sep 17, 2026
In-file parallelism Open

enhancement

huggingface/datatrove#74 · 3 comments · 2 reactions · 0 assignees · Indexed Sep 17, 2026
Flexibility in minhash dedup by index Open

huggingface/datatrove#110 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Support int32 in substring dedup Open

huggingface/datatrove#118 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Potential issues in substring dedup Open

bug

huggingface/datatrove#121 · 10 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Memory overhead in multiprocessing Open

huggingface/datatrove#161 · 8 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
PineconeWriter Open

huggingface/datatrove#165 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
OpensearchWriter Open

huggingface/datatrove#166 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Periodical logging of stats Open

huggingface/datatrove#171 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Dependency resolving issue installing from source Open

huggingface/datatrove#173 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Fastwarc reader Open

huggingface/datatrove#182 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Migrate word tokenizer download functions to process locked download Open

huggingface/datatrove#190 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Minhash mersenne hashing overflow issues. Open

huggingface/datatrove#198 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Memory overflow issue with long-context data using datatrove Open

huggingface/datatrove#204 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
JSONL loading slow when using megawarcs Open

huggingface/datatrove#206 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Exploring documents in different clusters Open

huggingface/datatrove#209 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Get list of duplicate docs id's from minhash dedup. Open

huggingface/datatrove#213 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Assign more cpu to single task to speed it up for local executor? Open

huggingface/datatrove#214 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Exact deduplication Open

huggingface/datatrove#216 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
minhash dedup causes local machine to hang. Open

huggingface/datatrove#222 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
I would like to get help from Datatrove enthusiasts regarding issues I'm facing while running the example script. Open

huggingface/datatrove#235 · 31 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Batched filter inputs? Open

huggingface/datatrove#237 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
datatrove fails to handle tasks >1k with slurm job arrays Open

huggingface/datatrove#238 · 25 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
LocalPipelineExecutor does not use cpu cores Open

huggingface/datatrove#240 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
how to postpone filter init till it's running Open

huggingface/datatrove#242 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Slurm Cluster Size for FineWeb Open

huggingface/datatrove#245 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
adding an automatic log file `tail` under slurm executor Open

huggingface/datatrove#246 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
solved: how to launch a slurm executor from an interactive slurm job Open

huggingface/datatrove#248 · 0 comments · 6 reactions · 0 assignees · Indexed Sep 17, 2026
How about addding custom word_tokenizers? Open

enhancement

huggingface/datatrove#254 · 0 comments · 2 reactions · 0 assignees · Indexed Sep 17, 2026
Is there a plan to process data in sft format Open

huggingface/datatrove#262 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Optimize parquet output for remote reading Open

huggingface/datatrove#263 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Incorrect Job ID Extraction on Clusters with Custom Slurm Output Open

huggingface/datatrove#265 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
How to look into the processed data? Open

huggingface/datatrove#266 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Filter very slow Open

huggingface/datatrove#270 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[FEATURE] CC Filter Open

huggingface/datatrove#283 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Naming Gopher's "max_non_alpha_words_ratio" Open

huggingface/datatrove#289 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Multi-node parallelism on slurm clusters Open

huggingface/datatrove#290 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
SLURM cannot achieve cross-node parallelism Open

huggingface/datatrove#292 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Does fineweb.py perform Element and paragraph level deduplication? Open

huggingface/datatrove#295 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Wrong stats in multi-node local executor Open

huggingface/datatrove#297 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Unexpected performance degradation behavior in minhash deduplication stage 2 Open

huggingface/datatrove#298 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Explanation of what exactly the various dedup strategies do Open

huggingface/datatrove#299 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
`buffer of a multiple of 68 bytes` Minhash bug Open

huggingface/datatrove#301 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
How to read text files Open

huggingface/datatrove#305 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Embedding-based decontamination Open

huggingface/datatrove#310 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
How do I assess whether the accuracy before and after dedup Open

huggingface/datatrove#312 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 17, 2026
FastText: newer model Open

huggingface/datatrove#321 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Unexpected behavior when using sentence_dedup with split_sentences=True Open

huggingface/datatrove#324 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026

huggingface/datatrove

Get new issues in your inbox