apache / apache/incubator-graphar

[discussion][java-info] dependencies choice for java library

Open
#738 6 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

### Describe the enhancement requested

We need to discuss the choice of dependency for working with Parquet and other file formats in the java-info library.

In my opinion, out-of-core computation is currently only implemented in the `C++` library, while other language bindings (like `Spark`) primarily need to support ETL operations.

Under this assumption, we should prefer **lightweight**, focused libraries that can handle Parquet efficiently without pulling in large dependencies. For example, using the [parquet-java library](https://github.com/apache/parquet-java) could be sufficient for our current needs.

However, if we plan to support **Gremlin** querying directly in the Java library in the future, we may need to implement out-of-core query capabilities , such as filtering and projection at read time. In that case, [Apache Arrow](https://arrow.apache.org/docs/java/index.html) becomes a more compelling choice.

Looking forward to your thoughts.

### Component(s)

Java

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reviewing the java-info library's current dependency setup and the planned Parquet and Gremlin requirements, then compare parquet-java with Apache Arrow. Done means the dependency choice and whether out-of-core filtering and projection are in scope are agreed and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.