apache / apache/arrow-java

[Java] Arrow native reader load next batch importing vector from c causing performance degrade

Open
#164 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
94
Forks
152
Avg merge
3d 16h
Merged PRs (30d)
11

Description

### Describe the bug, including details regarding any error messages, version, and platform.

it seems the native reader is first read the data into c vector, then import it into java format, causing significant performance degrade.

Screenshot 2023-09-14 at 15 14 07

I wonder if it is possible to read directly from c vector ?

### Component(s)

Java

Contributor guide

Open the contributing guide

Research direction

Start with the Java Arrow native reader's load-next-batch path and the reported performance degradation. Profile the current import from a C vector into Java and determine whether direct reading is feasible; done requires measured improvement without changing reader results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.