huggingface / huggingface/datatrove
AssertionError: Need to run with n_tasks = n_files. len(sequence_file)=1, len(sequence_file)=1, len(byte_range_file)=0
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Hi,
I want to do exact substring deduplication following the example in [examples/exact_substrings.py](https://github.com/huggingface/datatrove/blob/main/examples/exact_substrings.py).
But I got an error because `*.bytearange` files never be generated and I can't pass the assertion at line 219 in [exact_substrings.py](https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/dedup/exact_substrings.py).
Is this a bug or did I miss any steps?
Here is to reproduce my errors.
```
pipeline_1 = [
JsonlReader("data/00000.jsonl.gz"),
ESDatasetToSequence(output_folder="es/"),
]
pipeline_2 = [
ESMergeSequences(
data_folder="es",
tasks_stage_1=4,
)
]
executor_1: PipelineExecutor = LocalPipelineExecutor(pipeline=pipeline_1, workers=4, tasks=4)
executor_2: PipelineExecutor = LocalPipelineExecutor(pipeline=pipeline_2, workers=1, tasks=1)
print(executor_1.run())
print(executor_2.run())
pipeline_3 = [
JsonlReader("intermediate/"), # must be the same data that was passed to DatasetToSequence
ESRangeRemover(
sequence_folder=f"{os.getcwd()}/es/",
),
JsonlWriter("final-deduped-data"),
]
```
I got the following error:
```
Traceback (most recent call last):
File ".../lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.12/site-packages/datatrove/executor/local.py", line 76, in _launch_run_for_rank
return self._run_for_rank(rank, local_rank)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.12/site-packages/datatrove/executor/base.py", line 109, in _run_for_rank
raise e
File ".../lib/python3.12/site-packages/datatrove/executor/base.py", line 96, in _run_for_rank
deque(pipelined_data, maxlen=0)
File ".../lib/python3.12/site-packages/datatrove/pipeline/writers/disk_base.py", line 178, in run
for document in data:
File ".../lib/python3.12/site-packages/datatrove/pipeline/dedup/exact_substrings.py", line 320, in run
sequence_file, size_file = self.get_all_files(rank=self.rank, world_size=world_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.12/site-packages/datatrove/pipeline/dedup/exact_substrings.py", line 219, in get_all_files
assert all(
AssertionError: Need to run with n_tasks = n_files. len(sequence_file)=1, len(sequence_file)=1, len(byte_range_file)=0
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.