apache / apache/incubator-graphar
[discussion][java-info] dependencies choice for java library
- Dominant language
- C++
- Stars
- 373
- Forks
- 93
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 7
Description
### Describe the enhancement requested
We need to discuss the choice of dependency for working with Parquet and other file formats in the java-info library.
In my opinion, out-of-core computation is currently only implemented in the `C++` library, while other language bindings (like `Spark`) primarily need to support ETL operations.
Under this assumption, we should prefer **lightweight**, focused libraries that can handle Parquet efficiently without pulling in large dependencies. For example, using the [parquet-java library](https://github.com/apache/parquet-java) could be sufficient for our current needs.
However, if we plan to support **Gremlin** querying directly in the Java library in the future, we may need to implement out-of-core query capabilities , such as filtering and projection at read time. In that case, [Apache Arrow](https://arrow.apache.org/docs/java/index.html) becomes a more compelling choice.
Looking forward to your thoughts.
### Component(s)
Java
Contributor guide
Research direction
No files or tests are named. Start by reviewing the java-info library's current dependency setup and the planned Parquet and Gremlin requirements, then compare parquet-java with Apache Arrow. Done means the dependency choice and whether out-of-core filtering and projection are in scope are agreed and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100