huggingface / huggingface/datatrove
LocalPipelineExecutor does not use cpu cores
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
I am trying to process a CC dump using the LocalPipelineExecutor. My setup includes 6 files in the dump and a VM with 48 CPU cores. I run the code with 6 tasks and 48 workers, What I expect is that 48 cores should be utilized efficiently. Only 6 cores are actively processing the tasks.
Code:
```
executor = LocalPipelineExecutor(
pipeline=[
JsonlReader(data_folder=f"{cc_path}/{dump}",text_key="raw_content"),
URLFilter(),
GopherRepetitionFilter(language = "tr"),
GopherQualityFilter( language = "tr"),
C4QualityFilter(filter_no_terminal_punct=False,
language = "tr"),
C4BadWordsFilter(default_language = "tr"),
PIIFormatter(),
JsonlWriter(
output_folder=f"{out_path}/out-text-process-4/{dump}"
)
],
logging_dir="logs",
workers=48,
tasks=6
)
executor.run()
```
How can I use all cores to process data?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.