huggingface / huggingface/datasets
Large WebDataset: pyarrow.lib.ArrowCapacityError on load() even with streaming
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I am creating a large WebDataset-format dataset for sign language processing research, and a number of the videos are over 2GB. The instant I hit one of the shards with one of those videos, I get a ArrowCapacityError, even with streaming.
I made a config for the dataset that specifically includes just one problem shard, and the error triggers the instant you even run load_dataset(), even with streaming=True
```
ds = load_dataset("bible-nlp/sign-bibles", "ase_chronological_bible_translation_in_american_sign_language_119_introductions_and_passages_debugging_problem_shard", streaming=True, split="train")
```
This gives:
```
File "/opt/home/cleong/projects/semantic_and_visual_similarity/sign-bibles-dataset/sign_bibles_dataset/tasks/test_iteration.py", line 13, in iterate_keys
ds = load_dataset("bible-nlp/sign-bibles", language_subset, streaming=True, split="train")
File "/opt/home/cleong/envs/sign-bibles-dataset/lib/python3.13/site-packages/datasets/load.py", line 1409, in load_dataset
return builder_instance.as_streaming_dataset(split=split)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/home/cleong/envs/sign-bibles-dataset/lib/python3.13/site-packages/datasets/builder.py", line 1225, in as_streaming_dataset
splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/opt/home/cleong/envs/sign-bibles-dataset/lib/python3.13/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 88, in _split_generators
pa.Table.from_pylist(cast_to_python_objects([example], only_1d_for_numpy=True))
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 2046, in pyarrow.lib._Tabular.from_pylist
File "pyarrow/table.pxi", line 6431, in pyarrow.lib._from_pylist
File "pyarrow/table.pxi", line 4893, in pyarrow.lib.Table.from_arrays
File "pyarrow/table.pxi", line 1607, in pyarrow.lib._sanitize_arrays
File "pyarrow/table.pxi", line 1588, in pyarrow.lib._schema_from_arrays
File "pyarrow/array.pxi", line 375, in pyarrow.lib.array
File "pyarrow/array.pxi", line 45, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 3980158992
```
### Steps to reproduce the bug
```python
#!/usr/bin/env python
import argparse
from datasets import get_dataset_config_names, load_dataset
from tqdm import tqdm
from pyarrow.lib import ArrowCapacityError, ArrowInvalid
def iterate_keys(language_subset: str) -> None:
"""Iterate over all samples in the Sign Bibles dataset and print idx and sample key."""
# https://huggingface.co/docs/datasets/v4.0.0/en/package_reference/loading_methods#datasets.load_dataset
ds = load_dataset("bible-nlp/sign-bibles", language_subset, streaming=True, split="train")
print(f"\n==> Loaded dataset config '{language_subset}'")
idx = 0
estimated_shard_index = 0
samples_per_shard = 5
with tqdm(desc=f"{language_subset} samples") as pbar:
iterator = iter(ds)
while True:
try:
if idx % samples_per_shard == 0 and idx > 0: # 5 samples per shard: 0, 1, 2, 3, 4
print(f"Estimated Shard idx (starting at 0, {samples_per_shard}/shard): {estimated_shard_index}")
estimated_shard_index += 1
sample = next(iterator)
sample_key = sample.get("__key__", "missing-key")
print(f"[{language_subset}] idx={idx}, key={sample_key}")
idx += 1
pbar.update(1)
except StopIteration:
print(f"Finished iterating through {idx} samples of {language_subset}")
break
except (ArrowCapacityError, ArrowInvalid) as e:
print(f"PyArrow error on idx={idx}, config={language_subset}: {e}")
idx += 1
pbar.update(1)
continue
except KeyError as e:
print(f"Missing key error on idx={idx}, config={language_subset}: {e}")
idx += 1
pbar.update(1)
continue
def main():
configs = get_dataset_config_names("bible-nlp/sign-bibles")
print(f"Available configs: {configs}")
configs = [
"ase_chronological_bible_translation_in_american_sign_language_119_introductions_and_passages_debugging_problem_shard"
]
for language_subset in configs:
print(f"TESTING CONFIG {language_subset}")
iterate_keys(language_subset)
# try:
# except (ArrowCapacityError, ArrowInvalid) as e:
# print(f"PyArrow error at config level for {language_subset}: {e}")
# continue
# except RuntimeError as e:
# print(f"RuntimeError at config level for {language_subset}: {e}")
# continue
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Iterate through Sign Bibles dataset and print sample keys.")
args = parser.parse_args()
main()
```
### Expected behavior
I expect, when I load with streaming=True, that there should not be any data loaded or anything like that.
https://huggingface.co/docs/datasets/main/en/package_reference/loading_methods#datasets.load_dataset says that with streaming=true,
I did expect to have some trouble with large files, but that the streaming mode would not actually try to load them unless requested, e.g. with sample["mp4"]
>In the streaming case:
> Don’t download or cache anything. Instead, the dataset is lazily loaded and will be streamed on-the-fly when iterating on it.
### Environment info
Local setup: Conda environment on Ubuntu, pip list includes the following
datasets 4.0.0
pyarrow 20.0.0
Verified on Colab: https://colab.research.google.com/drive/1HdN8stlROWrLSYXUoNeV0vQ9pClhIVM8?usp=sharing, though there it crashes by using up all available RAM
Contributor guide
Assessment
This issue has not been assessed yet.