apache / apache/arrow

[Python][Parquet] Writing a parquet table fails with `segfault`

Open
#37,747 5 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When trying to export a portion of real survey data from our system, we encountered a segfault when invoking `pq.write_table`. Granted that there might be problem with data, the function still shouldn't fail in such a way, but rather with a message. Here's the code snippet that reproduces the issue, together with 4 relevant `npy` files. I wasn't able to reproduce it in an explicit manner by generating data manually (this is a slice of real data that produced this error).

Here's the code:

```
"""Repro the discovered bug in pq writer."""

import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

if __name__ == "__main__":
# --- Load data ---
inds = np.load("inds.npy")
subrefs = np.load("subrefs.npy")
values = np.load("values.npy")
offsets = np.load("offsets.npy")

# --- Create table with single MapArray ---
keys = pa.DictionaryArray.from_arrays(inds, subrefs)
var = pa.MapArray.from_arrays(offsets, keys, values)
tbl = pa.Table.from_arrays([var], names=["mapvar"])

# --- Try to write table to a parquet file (fails with segfault) ---
pq.write_table(tbl, "test.parquet")
```

The relevant `npy` files are zipped in the archive here attached:

[data.zip](https://github.com/apache/arrow/files/12623235/data.zip)

Lmk if you need any clarifications. I'll try generating a minimal example, but this is what I've got for now.

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.