huggingface / huggingface/datatrove
Assign more cpu to single task to speed it up for local executor?
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
I am using the local executor. My machine has 48 Cpus with 348 Ram. Any idea how to speed this up? Currently one single task (task=1, running for 1 warc.gz file, with size ~1g) takes half an hour. This is my executor code, borrowed from the fineweb example. Also, I have 200 warc.gz files to process. Is setting tasks = 200 the correct way?
```python
main_processing_executor = LocalPipelineExecutor(
pipeline=[
WarcReader(
f"tur_subsubset",
compression="gzip",
glob_pattern="*.warc.gz",
),
URLFilter(),
Trafilatura(favour_precision=True, timeout=10),
LanguageFilter(languages=(Languages.turkish)),
GopherRepetitionFilter(),
GopherQualityFilter(),
C4QualityFilter(filter_no_terminal_punct=False),
FineWebQualityFilter(),
JsonlWriter(f"{FILTERING_OUTPUT_PATH}/output/{DUMP_TO_PROCESS}"),
],
tasks=200,
workers=44,
logging_dir=f"{MAIN_OUTPUT_PATH}/logs/base_processing/{DUMP_TO_PROCESS}",
)
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names LocalPipelineExecutor and shows a configuration using tasks=200 and workers=44; start by reviewing how those executor settings schedule the WarcReader pipeline. Reproduce the reported single-file run with the stated configuration, then establish what supported configuration or documentation would clarify processing the 200 WARC files and measure whether the result improves throughput.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100