datatrove fails to handle tasks >1k with slurm job arrays

If I have more tasks than 1k, datatrove splits it into multiple job arrays 1k-each.

the first job array of 1k runs fine, the subsequent ones all fail

```
0: 2024-07-04 23:59:34.496 | ERROR | datatrove.executor.base:_run_for_rank:108 - list index out of range
0: Traceback (most recent call last):
0:
0: File "/env/lib/conda/ctx-shared/bin/launch_pickled_pipeline", line 8, in
0: sys.exit(main())
0: │ │ └
0: │ └
0: └
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/tools/launch_pickled_pipeline.py", line 18, in main
0: executor.run()
0: │ └
0: └
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/executor/slurm.py", line 180, in run
0: self._run_for_rank(rank)
0: │ │ └ 7113
0: │ └
0: └
0: > File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/executor/base.py", line 96, in _run_for_rank
0: deque(pipelined_data, maxlen=0)
0: │ └
0: └
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/writers/disk_base.py", line 176, in run
0: for document in data:
0: └
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/filters/base_filter.py", line 47, in run
0: for doc in data:
0: └
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/readers/huggingface.py", line 96, in run
0: shard = self._get_dataset_shard(ds, rank, world_size)
0: │ │ │ │ └ 100000
0: │ │ │ └ 7113
0: │ │ └ IterableDataset({
0: │ │ features: ['text', 'id', 'dump', 'url', 'date', 'file_path', 'language', 'language_score', 'token_count...
0: │ └
0: └ 📖 - READER: 🤗 HuggingFace
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/readers/huggingface.py", line 69, in _get_dataset_shard
0: ex_iterable = dst._ex_iterable.shard_data_sources(rank, world_size)
0: │ │ │ │ └ 100000
0: │ │ │ └ 7113
0: │ │ └
0: │ └
0: └ IterableDataset({
0: features: ['text', 'id', 'dump', 'url', 'date', 'file_path', 'language', 'language_score', 'token_count...
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 298, in shard_data_sources
0: requested_gen_kwargs = _merge_gen_kwargs([gen_kwargs_list[i] for i in shard_indices])
0: │ │ └ []
0: │ └ [{'files': []}, {'files': [
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datasets/utils/sharding.py", line 76, in _merge_gen_kwargs
0: for key in gen_kwargs_list[0]
0: └ []
0:
0: IndexError: list index out of range
0: Traceback (most recent call last):
0: File "/env/lib/conda/ctx-shared/bin/launch_pickled_pipeline", line 8, in
0: sys.exit(main())
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/tools/launch_pickled_pipeline.py", line 18, in main
0: executor.run()
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/executor/slurm.py", line 180, in run
0: self._run_for_rank(rank)
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/executor/base.py", line 109, in _run_for_rank
0: raise e
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/executor/base.py", line 96, in _run_for_rank
0: deque(pipelined_data, maxlen=0)
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/writers/disk_base.py", line 176, in run
0: for document in data:
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/filters/base_filter.py", line 47, in run
0: for doc in data:
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/readers/huggingface.py", line 96, in run
0: shard = self._get_dataset_shard(ds, rank, world_size)
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datatrove/pipeline/readers/huggingface.py", line 69, in _get_dataset_shard
0: ex_iterable = dst._ex_iterable.shard_data_sources(rank, world_size)
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 298, in shard_data_sources
0: requested_gen_kwargs = _merge_gen_kwargs([gen_kwargs_list[i] for i in shard_indices])
0: File "/env/lib/conda/ctx-shared/lib/python3.10/site-packages/datasets/utils/sharding.py", line 76, in _merge_gen_kwargs
0: for key in gen_kwargs_list[0]
0: IndexError: list index out of range
srun: error: dojo-a3-ghpc-41: task 0: Exited with exit code 1
```

This failing behavior is consistent

This is with `datatrove@main` - I can't use the official release as it doesn't support datasets streaming.

This is just doing a slightly modified FineWeb filter from the example.

Reproduce the failure with more than 1,000 tasks using the modified FineWeb filter example. Start with datatrove/executor/slurm.py and datatrove/pipeline/readers/huggingface.py, especially HuggingFaceDatasetReader._get_dataset_shard and the reported sharding traceback. Done means subsequent Slurm job arrays complete without the IndexError.

Written by the indexing model from the issue text.