apache / apache/arrow

[Python] Table contains dictionary encoding with negative index

Open
#37,732 4 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

OS: RHEL 8.3
Language: Python 3.8.15
PyArrow Version: 9.0.0

I have a script that streams data to a file using ` pa.ipc.new_stream(path, reader.schema)`, and in a separate process, reading from that file using:
```
with pa.ipc.open_stream(path) as reader:
batches = list(reader)
df = pa.Table.from_batches(batches).to_pandas()
```
which fails with
```
pyarrow.lib.ArrowIndexError: Index -241 out of bounds
```

Inexplicably, in some cases it looks like, in a specific column with a dictionary encoding, one of the rows has a dictionary index of -241, even though the dictionary itself consists of a single value. Is this standard file corruption, a bug, or am I doing something wrong in the process writing/reading described above?

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.