apache / apache/arrow

[Python] Data is partially lost when chunks have different columns

Open
#47,911 1 comment 2 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Image

1. I construct an arrow array from big native python list of dicts, that forces arrow to create a chunked array
2. Array is chunked in such a way that some colum(s) are only present in one chunk
3. That forces some method not to work at all (i.e. `table.to_pandas`) while others (`table.to_reader`) produce incorrect data (they say columns are `None` even when the value is actually set)

```python
import pyarrow as pa

pa.show_versions()

data = [
{"key_1": "abcd" * 100_000, "pk": i} # large enough rows to force chunking. replace 100_000 with 10 and everything below will pass
for i in range(10_000)
]

data.append({"key_1": "abcd", "pk": 10_000, "extra_key": "value"})

t = pa.array(data)
print(t.to_pandas().iloc[-1])

table = pa.Table.from_struct_array(t)

# print(table.to_pandas().iloc[-1]) # ← this fails with:
# ValueError: Shape of passed values is (10002, 3), indices imply (10001, 3)

r = table.to_reader()

vals = set()

while True:
try:
batch = r.read_next_batch()
vals.update(batch.column("extra_key").to_pylist())

except StopIteration:
break

print("\nUnique values in 'extra_key':", vals)
# assert vals == {"value", None} # this fails because vals is { None }
```

```
Package kind : python-wheel-macos
Arrow C++ library version : 21.0.0
Arrow C++ compiler : AppleClang 15.0.0.15000309
Arrow C++ compiler flags : -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wno-unknown-warning-option -Wno-pass-failed
Arrow C++ git revision : ee4d09ebef61c663c1efbfa4c18e518a03b798be
Arrow C++ git description : apache-arrow-21.0.0-rc6
Arrow C++ build type : release
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.