apache / apache/arrow

[C++][Python] pyarrow.csv.write_csv crashes when writing tables containing FixedSizeBinaryArray

Open
#35,901 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Suppose we have a large array of binary strings
```
import numpy as np
import pyarrow as pa
import pyarrow.csv as pcsv

nparr = np.frombuffer(np.random.randint(65, 91, int(4E8), 'u1'), 'S4')
```
We want to construct a FixedSizedBinaryArray and Table
```
fixedarr = pa.array(nparr, pa.binary(4))
fixedtable = pa.Table.from_arrays([fixedarr], names=['fixedsize'])
```

Alternatively, construct a BinaryArray and Table
```
binarr = pa.array(nparr, pa.binary())
bintable = pa.Table.from_arrays([binarr], names=['binary'])
```
Setting up the csv write option
```
csvoption = pcsv.WriteOptions(include_header=False, batch_size=2048, delimiter='|', quoting_style='none')
```
Writing the table containing BinaryArray works
```
pcsv.write_csv(bintable, 'binary.csv', write_options=csvoption)
```

But writing the table containing FixedSizedBinaryArray crashes after first `batch_size`
```
pcsv.write_csv(fixedtable, 'fixedsize.csv', write_options=csvoption)
```
Note both tables are compatible with `pyarrow.parquet.write_table` though.

**Environment**:
Windows 10
pyarrow 12.0.0
Python 3.10.10

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure through the pyarrow.csv.write_csv entry point using the provided BinaryArray and FixedSizeBinaryArray examples, with batch_size set to 2048. Compare the two write paths and verify that FixedSizeBinaryArray tables complete without crashing and produce CSV output.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.