Seeking EOF of GCS File fails via Python
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Bug in seek(0,2) when trying to read bytes to EOF with GCS for PyArrow 10.0.x
MWE:
create test file:
```
#test.py
x = 1 +2
def func(x):
return x + 1
```
push to GCS
`gsutil cp test.py gs://bucket/`
run:
```
from pyarrow.fs import GcsFileSystem
fs = GcsFileSystem()
f = fs.open_input_file("bucket/test.py")
f.size()
#Out[9]: 48
f.seek(0,2) # WHENCE EOF = 2
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[10], line 1
----> 1 f.seek(0,2)
File ~/.virtualenvs/venv-test/lib/python3.10/site-packages/pyarrow/io.pxi:323, in pyarrow.lib.NativeFile.seek()
File ~/.virtualenvs/venv-test/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()
ArrowInvalid: google::cloud::Status(OUT_OF_RANGE: Permanent error ReadObjectNotWrapped: InvalidRangeThe requested range cannot be satisfied.bytes=10448-)
```
I expect this is because we use the `gcs::Generation` which makes a byte range call to gcs, and by HTTP byte-range RFC, bytes are inclusive, and doesn't contain an EOF byte
### Component(s)
C++, Python
Contributor guide
Assessment
This issue has not been assessed yet.