[VL] Align standalone Arrow native and Java versions
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
PR #12625 moves the vcpkg build path to native Arrow 18.1.0, but the standalone build paths are not consistently aligned:
- `dev/build-arrow.sh` defaults to Arrow 15.0.0.
- `dev/build-arrow-18.sh` provides Arrow 18.1.0 only for the current ppc64le Spark 4.0 special case.
- Spark 3.x Maven profiles use Arrow Java 15.0.0, while Spark 4.x profiles use Arrow Java 18.1.0.
- ppc64le builds Arrow Java/JNI artifacts locally, so the native and Java versions must match.
A direct replacement of 15.0.0 with 18.1.0 is unsafe because the standalone Arrow patches need to be reviewed or updated for Arrow 18, and upgrading ppc64le native Arrow without coordinating the Maven Arrow version can produce incompatible JNI artifacts.
The follow-up should:
- Define the intended native/Java Arrow version mapping for Spark 3.x and Spark 4.x standalone builds.
- Update or remove the standalone Arrow patches as needed for Arrow 18.
- Consolidate `build-arrow.sh` and `build-arrow-18.sh` if practical.
- Ensure locally built ppc64le Arrow C Data and Dataset JNI artifacts match the Maven `arrow.version`.
- Validate the affected standalone builds and PPC CI paths.
Gluten version: main branch
Generated-by: GitHub Copilot CLI 1.0.78-2
Contributor guide
Research direction
Inspect dev/build-arrow.sh and dev/build-arrow-18.sh, the Spark 3.x and 4.x Maven profiles, standalone Arrow patches, and the PPC CI paths. First compare the native and Java version settings and patch compatibility; done means the intended mapping is documented in the build configuration, JNI artifacts match Maven arrow.version, and affected standalone and PPC builds pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, ci-cd
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100