[C++][Python] pyarrow.csv.write_csv crashes when writing tables containing FixedSizeBinaryArray
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Suppose we have a large array of binary strings
```
import numpy as np
import pyarrow as pa
import pyarrow.csv as pcsv
nparr = np.frombuffer(np.random.randint(65, 91, int(4E8), 'u1'), 'S4')
```
We want to construct a FixedSizedBinaryArray and Table
```
fixedarr = pa.array(nparr, pa.binary(4))
fixedtable = pa.Table.from_arrays([fixedarr], names=['fixedsize'])
```
Alternatively, construct a BinaryArray and Table
```
binarr = pa.array(nparr, pa.binary())
bintable = pa.Table.from_arrays([binarr], names=['binary'])
```
Setting up the csv write option
```
csvoption = pcsv.WriteOptions(include_header=False, batch_size=2048, delimiter='|', quoting_style='none')
```
Writing the table containing BinaryArray works
```
pcsv.write_csv(bintable, 'binary.csv', write_options=csvoption)
```
But writing the table containing FixedSizedBinaryArray crashes after first `batch_size`
```
pcsv.write_csv(fixedtable, 'fixedsize.csv', write_options=csvoption)
```
Note both tables are compatible with `pyarrow.parquet.write_table` though.
**Environment**:
Windows 10
pyarrow 12.0.0
Python 3.10.10
### Component(s)
Python
Contributor guide
Research direction
Reproduce the failure through the pyarrow.csv.write_csv entry point using the provided BinaryArray and FixedSizeBinaryArray examples, with batch_size set to 2048. Compare the two write paths and verify that FixedSizeBinaryArray tables complete without crashing and produce CSV output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100