[Python] Data is partially lost when chunks have different columns
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
1. I construct an arrow array from big native python list of dicts, that forces arrow to create a chunked array
2. Array is chunked in such a way that some colum(s) are only present in one chunk
3. That forces some method not to work at all (i.e. `table.to_pandas`) while others (`table.to_reader`) produce incorrect data (they say columns are `None` even when the value is actually set)
```python
import pyarrow as pa
pa.show_versions()
data = [
{"key_1": "abcd" * 100_000, "pk": i} # large enough rows to force chunking. replace 100_000 with 10 and everything below will pass
for i in range(10_000)
]
data.append({"key_1": "abcd", "pk": 10_000, "extra_key": "value"})
t = pa.array(data)
print(t.to_pandas().iloc[-1])
table = pa.Table.from_struct_array(t)
# print(table.to_pandas().iloc[-1]) # ← this fails with:
# ValueError: Shape of passed values is (10002, 3), indices imply (10001, 3)
r = table.to_reader()
vals = set()
while True:
try:
batch = r.read_next_batch()
vals.update(batch.column("extra_key").to_pylist())
except StopIteration:
break
print("\nUnique values in 'extra_key':", vals)
# assert vals == {"value", None} # this fails because vals is { None }
```
```
Package kind : python-wheel-macos
Arrow C++ library version : 21.0.0
Arrow C++ compiler : AppleClang 15.0.0.15000309
Arrow C++ compiler flags : -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wno-unknown-warning-option -Wno-pass-failed
Arrow C++ git revision : ee4d09ebef61c663c1efbfa4c18e518a03b798be
Arrow C++ git description : apache-arrow-21.0.0-rc6
Arrow C++ build type : release
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.