huggingface / huggingface/datatrove

Assign more cpu to single task to speed it up for local executor?

Open
#214 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

I am using the local executor. My machine has 48 Cpus with 348 Ram. Any idea how to speed this up? Currently one single task (task=1, running for 1 warc.gz file, with size ~1g) takes half an hour. This is my executor code, borrowed from the fineweb example. Also, I have 200 warc.gz files to process. Is setting tasks = 200 the correct way?

```python
main_processing_executor = LocalPipelineExecutor(
pipeline=[
WarcReader(
f"tur_subsubset",
compression="gzip",
glob_pattern="*.warc.gz",
),
URLFilter(),
Trafilatura(favour_precision=True, timeout=10),
LanguageFilter(languages=(Languages.turkish)),
GopherRepetitionFilter(),
GopherQualityFilter(),
C4QualityFilter(filter_no_terminal_punct=False),
FineWebQualityFilter(),
JsonlWriter(f"{FILTERING_OUTPUT_PATH}/output/{DUMP_TO_PROCESS}"),
],

tasks=200,
workers=44,
logging_dir=f"{MAIN_OUTPUT_PATH}/logs/base_processing/{DUMP_TO_PROCESS}",
)

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names LocalPipelineExecutor and shows a configuration using tasks=200 and workers=44; start by reviewing how those executor settings schedule the WarcReader pipeline. Reproduce the reported single-file run with the stated configuration, then establish what supported configuration or documentation would clarify processing the 200 WARC files and measure whether the result improves throughput.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.