apache / apache/arrow

orc.write_table performance degradation when writing table with structs

Open
#35,840 0 comments 3 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hi. We observe `orc.write_table` performance degradation compared to `parquet.write_table` when table has structs. When we write same amount of data as a single flattened array - orc writer gives similar write time as parquet. But we need to have these structs to be able to upload multidimensional arrays to Bigquery securing their original dimensionality.

```
pyarrow versions tested: 9.0.0, 11.0.0, 12.0.0
os tested: MAC M1, Linux
```

Reproducible example:

WRITING SRUCTS:
```
import time
import numpy as np
import pyarrow as pa
from pyarrow import orc, parquet, compute as pa_compute
from matplotlib import pyplot as plt

batch_sizes = [8, 16, 32, 64]

orc_writetime, parquet_writetime = [], []

for batch_size in batch_sizes:

batch = np.random.rand(batch_size, 384, 400)
array = pa.array(batch.reshape(-1))

for dim in reversed(batch.shape):
array = pa.FixedSizeListArray.from_arrays(array, dim)
array = pa_compute.make_struct(array, field_names=["values"])

table = pa.Table.from_arrays([array], names=["test"])

start = time.time()
with pa.BufferOutputStream() as buffer:
orc.write_table(table, buffer, compression="snappy")
orc_writetime.append(time.time() - start)

start = time.time()
with pa.BufferOutputStream() as buffer:
parquet.write_table(table, buffer, compression="snappy")
parquet_writetime.append(time.time() - start)

plt.figure(figsize=(20,5))
plt.plot(orc_writetime)
plt.plot(parquet_writetime)
plt.xticks(range(len(batch_sizes)), batch_sizes)
plt.title("Writing structs experiment")
plt.xlabel("batch_size")
plt.ylabel("write time, seconds")
plt.legend(["orc", "parquet"])
plt.grid()
```
![image](https://github.com/apache/arrow/assets/9593371/18f2f2b7-51bb-40aa-9a5a-923b1231300e)

WRITING ARRAYS
```
orc_writetime, parquet_writetime = [], []

for batch_size in batch_sizes:

batch = np.random.rand(batch_size, 384, 400)
array = pa.array(batch.reshape(-1))

table = pa.Table.from_arrays([array], names=["test"])

start = time.time()
with pa.BufferOutputStream() as buffer:
orc.write_table(table, buffer, compression="snappy")
orc_writetime.append(time.time() - start)

start = time.time()
with pa.BufferOutputStream() as buffer:
parquet.write_table(table, buffer, compression="snappy")
parquet_writetime.append(time.time() - start)

plt.figure(figsize=(20,5))
plt.plot(orc_writetime)
plt.plot(parquet_writetime)
plt.xticks(range(len(batch_sizes)), batch_sizes)
plt.title("Writing arrays experiment")
plt.xlabel("batch_size")
plt.ylabel("write time, seconds")
plt.legend(["orc", "parquet"])
plt.grid()
```
![image](https://github.com/apache/arrow/assets/9593371/21954b9a-a53c-4195-9bfb-8c4e04f90e0d)

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the reproducible Python example comparing orc.write_table and parquet.write_table for nested FixedSizeListArray structs, then investigate the ORC writer path used by pyarrow. Confirm the performance gap across the listed pyarrow versions and platforms; done means struct-containing tables write without the reported degradation while preserving their nested dimensionality.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.