apache / apache/arrow

[Python] Reading partial data/first block hangs on some cloud filesystems

Open
#43,497 17 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Take the following example using a publicly available dataset:

```python
import gcsfs
from pyarrow.dataset import dataset

# without fsspec filesystem, get segmentation fault
fs = None
# with fsspec filesystem, hangs and never finishes
# fs = gcsfs.GCSFileSystem()

uri = "gs://datachain-demo/laion-aesthetics-csv/laion_aesthetics_1024_33M_1.csv"
ds = dataset(uri, format="csv", filesystem=fs)
print(ds.head(5))
```

As noted in the comments, depending on which filesystem is passed, it will either hang indefinitely or hit a segmentation fault. Strangely, s3 paths work (don't hang or fail) with the pyarrow filesystem but hang with the fsspec s3fs filesystem. Local paths work with either the pyarrow or fsspec filesystem.

Other findings:
- Similar operations like `ds.take()` and `next(ds.to_batches())` have the same behavior as `ds.head()`
- `ds.head(use_threads=False)` completes successfully with any filesystem but takes much longer

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.