Streaming: Pyarrow is 15 times faster than Arrow.jl
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
I have an `.arrow` file generated with `pyarrow` whose schema is the following:
```
input: struct[512], high: fixed_size_list[512], low: fixed_size_list[512], close: fixed_size_list[512]> not null
child 0, open: fixed_size_list[512]
child 0, item: float
child 1, high: fixed_size_list[512]
child 0, item: float
child 2, low: fixed_size_list[512]
child 0, item: float
child 3, close: fixed_size_list[512]
child 0, item: float
```
With `pyarrow`, I load and iterate over records with the following:
```python
with pa.memory_map('arraydata.arrow', 'r') as source:
loaded_arrays = pa.ipc.open_file(source).read_all()
a = 0
for batch in loaded_arrays.to_batches():
for input_candles in batch["input"]:
a += 1
```
Iterating over my example file (~10,000 lines) takes 210 ms.
In julia, I load and iterate over the same file with the following:
```julia
stream = Arrow.Stream("./arraydata.arrow")
function bench_iteration(stream)
a = 0
for batch in stream
for sample in batch.input
a += 1
end
end
end
@btime bench_iteration($stream)
```
```
3.169 s (25272097 allocations: 1.70 GiB)
```
Iterating over records takes 15 more time with `Arrow.jl`. Am I doing something wrong?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the comparison using Arrow.Stream("./arraydata.arrow"), the provided bench_iteration function, and the matching pyarrow IPC iteration. Inspect the Arrow.Stream batch and nested input iteration paths, then measure allocations and elapsed time; done means explaining or correcting the large performance gap on the supplied .arrow file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia, python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100