apache / apache/gluten

[VL] Align standalone Arrow native and Java versions

Open
#12,673 0 comments 0 reactions 0 assignees View on GitHub
enhancement velox backend
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

PR #12625 moves the vcpkg build path to native Arrow 18.1.0, but the standalone build paths are not consistently aligned:

- `dev/build-arrow.sh` defaults to Arrow 15.0.0.
- `dev/build-arrow-18.sh` provides Arrow 18.1.0 only for the current ppc64le Spark 4.0 special case.
- Spark 3.x Maven profiles use Arrow Java 15.0.0, while Spark 4.x profiles use Arrow Java 18.1.0.
- ppc64le builds Arrow Java/JNI artifacts locally, so the native and Java versions must match.

A direct replacement of 15.0.0 with 18.1.0 is unsafe because the standalone Arrow patches need to be reviewed or updated for Arrow 18, and upgrading ppc64le native Arrow without coordinating the Maven Arrow version can produce incompatible JNI artifacts.

The follow-up should:

- Define the intended native/Java Arrow version mapping for Spark 3.x and Spark 4.x standalone builds.
- Update or remove the standalone Arrow patches as needed for Arrow 18.
- Consolidate `build-arrow.sh` and `build-arrow-18.sh` if practical.
- Ensure locally built ppc64le Arrow C Data and Dataset JNI artifacts match the Maven `arrow.version`.
- Validate the affected standalone builds and PPC CI paths.

Gluten version: main branch

Generated-by: GitHub Copilot CLI 1.0.78-2

Contributor guide

Open the contributing guide

Research direction

Inspect dev/build-arrow.sh and dev/build-arrow-18.sh, the Spark 3.x and 4.x Maven profiles, standalone Arrow patches, and the PPC CI paths. First compare the native and Java version settings and patch compatibility; done means the intended mapping is documented in the build configuration, JNI artifacts match Maven arrow.version, and affected standalone and PPC builds pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, ci-cd
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.