[Python] Table contains dictionary encoding with negative index
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
OS: RHEL 8.3
Language: Python 3.8.15
PyArrow Version: 9.0.0
I have a script that streams data to a file using ` pa.ipc.new_stream(path, reader.schema)`, and in a separate process, reading from that file using:
```
with pa.ipc.open_stream(path) as reader:
batches = list(reader)
df = pa.Table.from_batches(batches).to_pandas()
```
which fails with
```
pyarrow.lib.ArrowIndexError: Index -241 out of bounds
```
Inexplicably, in some cases it looks like, in a specific column with a dictionary encoding, one of the rows has a dictionary index of -241, even though the dictionary itself consists of a single value. Is this standard file corruption, a bug, or am I doing something wrong in the process writing/reading described above?
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.