apache / apache/arrow-julia

Streaming: Pyarrow is 15 times faster than Arrow.jl

Open
#334 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I have an `.arrow` file generated with `pyarrow` whose schema is the following:
```
input: struct[512], high: fixed_size_list[512], low: fixed_size_list[512], close: fixed_size_list[512]> not null
child 0, open: fixed_size_list[512]
child 0, item: float
child 1, high: fixed_size_list[512]
child 0, item: float
child 2, low: fixed_size_list[512]
child 0, item: float
child 3, close: fixed_size_list[512]
child 0, item: float
```

With `pyarrow`, I load and iterate over records with the following:
```python
with pa.memory_map('arraydata.arrow', 'r') as source:
loaded_arrays = pa.ipc.open_file(source).read_all()

a = 0
for batch in loaded_arrays.to_batches():
for input_candles in batch["input"]:
a += 1
```
Iterating over my example file (~10,000 lines) takes 210 ms.

In julia, I load and iterate over the same file with the following:

```julia
stream = Arrow.Stream("./arraydata.arrow")

function bench_iteration(stream)
a = 0
for batch in stream
for sample in batch.input
a += 1
end
end
end

@btime bench_iteration($stream)
```

```
3.169 s (25272097 allocations: 1.70 GiB)
```

Iterating over records takes 15 more time with `Arrow.jl`. Am I doing something wrong?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the comparison using Arrow.Stream("./arraydata.arrow"), the provided bench_iteration function, and the matching pyarrow IPC iteration. Inspect the Arrow.Stream batch and nested input iteration paths, then measure allocations and elapsed time; done means explaining or correcting the large performance gap on the supplied .arrow file.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia, python
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.