huggingface / huggingface/datatrove

LocalPipelineExecutor does not use cpu cores

Open
#240 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

I am trying to process a CC dump using the LocalPipelineExecutor. My setup includes 6 files in the dump and a VM with 48 CPU cores. I run the code with 6 tasks and 48 workers, What I expect is that 48 cores should be utilized efficiently. Only 6 cores are actively processing the tasks.

Code:
```
executor = LocalPipelineExecutor(
pipeline=[
JsonlReader(data_folder=f"{cc_path}/{dump}",text_key="raw_content"),
URLFilter(),
GopherRepetitionFilter(language = "tr"),
GopherQualityFilter( language = "tr"),
C4QualityFilter(filter_no_terminal_punct=False,
language = "tr"),
C4BadWordsFilter(default_language = "tr"),
PIIFormatter(),
JsonlWriter(
output_folder=f"{out_path}/out-text-process-4/{dump}"
)
],
logging_dir="logs",
workers=48,
tasks=6
)
executor.run()
```

How can I use all cores to process data?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.