apache / apache/gluten

[CORE] Source folder control for different Spark versions in all Maven modules

Open
#9,995 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

## Background

In CH and VL backlends, we started to have the demand to selectively compile prod / test code for Iceberg with different Spark versions, the code is under specific handlings of Maven configurations in `backends-velox/pom.xml` and `backends-clickhouse/pom.xml`. Though the Maven behaviours are different: In CH backend, some code won't compile if mismatched Spark version is set in Maven when Iceberg is enabled. In VL, we have added [some source exclusions](https://github.com/apache/incubator-gluten/blob/ca2254eda51a37b4dae7f8672735f31896ba5442/backends-velox/pom.xml#L222-L225) to satisfy Maven, however the Intellij IDEA IDE doesn't always recognize this trick and may report errors on that.

## Proposed Changes

To solve the issue, similar to our previous https://github.com/apache/incubator-gluten/issues/8208, we could add the global source folders for finer grained code activation control for different Spark versions.

These folders could be:

1. src-spark32, src-spark33, ...
2. src-delta-spark32, src-delta-spark33, ...
3. src-iceberg-spark32, src-iceberg-spark33, ...
4. ...

## Related previous PRs

https://github.com/apache/incubator-gluten/pull/8800
https://github.com/apache/incubator-gluten/pull/8256

Contributor guide

Open the contributing guide

Research direction

Start by comparing the Maven configurations in backends-velox/pom.xml and backends-clickhouse/pom.xml, then review issue 8208 and PRs 8800 and 8256 for the earlier approach. Done means the Maven modules can selectively activate source folders for different Spark and integration versions without mismatched compilation or IntelliJ IDEA errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.