orc.write_table performance degradation when writing table with structs
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hi. We observe `orc.write_table` performance degradation compared to `parquet.write_table` when table has structs. When we write same amount of data as a single flattened array - orc writer gives similar write time as parquet. But we need to have these structs to be able to upload multidimensional arrays to Bigquery securing their original dimensionality.
```
pyarrow versions tested: 9.0.0, 11.0.0, 12.0.0
os tested: MAC M1, Linux
```
Reproducible example:
WRITING SRUCTS:
```
import time
import numpy as np
import pyarrow as pa
from pyarrow import orc, parquet, compute as pa_compute
from matplotlib import pyplot as plt
batch_sizes = [8, 16, 32, 64]
orc_writetime, parquet_writetime = [], []
for batch_size in batch_sizes:
batch = np.random.rand(batch_size, 384, 400)
array = pa.array(batch.reshape(-1))
for dim in reversed(batch.shape):
array = pa.FixedSizeListArray.from_arrays(array, dim)
array = pa_compute.make_struct(array, field_names=["values"])
table = pa.Table.from_arrays([array], names=["test"])
start = time.time()
with pa.BufferOutputStream() as buffer:
orc.write_table(table, buffer, compression="snappy")
orc_writetime.append(time.time() - start)
start = time.time()
with pa.BufferOutputStream() as buffer:
parquet.write_table(table, buffer, compression="snappy")
parquet_writetime.append(time.time() - start)
plt.figure(figsize=(20,5))
plt.plot(orc_writetime)
plt.plot(parquet_writetime)
plt.xticks(range(len(batch_sizes)), batch_sizes)
plt.title("Writing structs experiment")
plt.xlabel("batch_size")
plt.ylabel("write time, seconds")
plt.legend(["orc", "parquet"])
plt.grid()
```

WRITING ARRAYS
```
orc_writetime, parquet_writetime = [], []
for batch_size in batch_sizes:
batch = np.random.rand(batch_size, 384, 400)
array = pa.array(batch.reshape(-1))
table = pa.Table.from_arrays([array], names=["test"])
start = time.time()
with pa.BufferOutputStream() as buffer:
orc.write_table(table, buffer, compression="snappy")
orc_writetime.append(time.time() - start)
start = time.time()
with pa.BufferOutputStream() as buffer:
parquet.write_table(table, buffer, compression="snappy")
parquet_writetime.append(time.time() - start)
plt.figure(figsize=(20,5))
plt.plot(orc_writetime)
plt.plot(parquet_writetime)
plt.xticks(range(len(batch_sizes)), batch_sizes)
plt.title("Writing arrays experiment")
plt.xlabel("batch_size")
plt.ylabel("write time, seconds")
plt.legend(["orc", "parquet"])
plt.grid()
```

### Component(s)
Python
Contributor guide
Research direction
Start with the reproducible Python example comparing orc.write_table and parquet.write_table for nested FixedSizeListArray structs, then investigate the ORC writer path used by pyarrow. Confirm the performance gap across the listed pyarrow versions and platforms; done means struct-containing tables write without the reported degradation while preserving their nested dimensionality.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100