apache / apache/arrow

Seeking EOF of GCS File fails via Python

Open
#38,854 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Bug in seek(0,2) when trying to read bytes to EOF with GCS for PyArrow 10.0.x

MWE:
create test file:
```
#test.py
x = 1 +2
def func(x):
return x + 1
```
push to GCS
`gsutil cp test.py gs://bucket/`

run:
```
from pyarrow.fs import GcsFileSystem

fs = GcsFileSystem()
f = fs.open_input_file("bucket/test.py")

f.size()
#Out[9]: 48

f.seek(0,2) # WHENCE EOF = 2
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[10], line 1
----> 1 f.seek(0,2)

File ~/.virtualenvs/venv-test/lib/python3.10/site-packages/pyarrow/io.pxi:323, in pyarrow.lib.NativeFile.seek()

File ~/.virtualenvs/venv-test/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: google::cloud::Status(OUT_OF_RANGE: Permanent error ReadObjectNotWrapped: InvalidRangeThe requested range cannot be satisfied.bytes=10448-)
```

I expect this is because we use the `gcs::Generation` which makes a byte range call to gcs, and by HTTP byte-range RFC, bytes are inclusive, and doesn't contain an EOF byte

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.