huggingface / huggingface/datatrove

AssertionError: Need to run with n_tasks = n_files. len(sequence_file)=1, len(sequence_file)=1, len(byte_range_file)=0

Open
#357 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

Hi,
I want to do exact substring deduplication following the example in [examples/exact_substrings.py](https://github.com/huggingface/datatrove/blob/main/examples/exact_substrings.py).
But I got an error because `*.bytearange` files never be generated and I can't pass the assertion at line 219 in [exact_substrings.py](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/dedup/exact_substrings.py).

Is this a bug or did I miss any steps?

Here is to reproduce my errors.
```
pipeline_1 = [
JsonlReader("data/00000.jsonl.gz"),
ESDatasetToSequence(output_folder="es/"),
]

pipeline_2 = [
ESMergeSequences(
data_folder="es",
tasks_stage_1=4,
)
]

executor_1: PipelineExecutor = LocalPipelineExecutor(pipeline=pipeline_1, workers=4, tasks=4)

executor_2: PipelineExecutor = LocalPipelineExecutor(pipeline=pipeline_2, workers=1, tasks=1)

print(executor_1.run())
print(executor_2.run())

pipeline_3 = [
JsonlReader("intermediate/"), # must be the same data that was passed to DatasetToSequence
ESRangeRemover(
sequence_folder=f"{os.getcwd()}/es/",
),
JsonlWriter("final-deduped-data"),
]
```

I got the following error:
```
Traceback (most recent call last):
File ".../lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.12/site-packages/datatrove/executor/local.py", line 76, in _launch_run_for_rank
return self._run_for_rank(rank, local_rank)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.12/site-packages/datatrove/executor/base.py", line 109, in _run_for_rank
raise e
File ".../lib/python3.12/site-packages/datatrove/executor/base.py", line 96, in _run_for_rank
deque(pipelined_data, maxlen=0)
File ".../lib/python3.12/site-packages/datatrove/pipeline/writers/disk_base.py", line 178, in run
for document in data:
File ".../lib/python3.12/site-packages/datatrove/pipeline/dedup/exact_substrings.py", line 320, in run
sequence_file, size_file = self.get_all_files(rank=self.rank, world_size=world_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.12/site-packages/datatrove/pipeline/dedup/exact_substrings.py", line 219, in get_all_files
assert all(
AssertionError: Need to run with n_tasks = n_files. len(sequence_file)=1, len(sequence_file)=1, len(byte_range_file)=0
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.