[CORE] Source folder control for different Spark versions in all Maven modules
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
## Background
In CH and VL backlends, we started to have the demand to selectively compile prod / test code for Iceberg with different Spark versions, the code is under specific handlings of Maven configurations in `backends-velox/pom.xml` and `backends-clickhouse/pom.xml`. Though the Maven behaviours are different: In CH backend, some code won't compile if mismatched Spark version is set in Maven when Iceberg is enabled. In VL, we have added [some source exclusions](https://github.com/apache/incubator-gluten/blob/ca2254eda51a37b4dae7f8672735f31896ba5442/backends-velox/pom.xml#L222-L225) to satisfy Maven, however the Intellij IDEA IDE doesn't always recognize this trick and may report errors on that.
## Proposed Changes
To solve the issue, similar to our previous https://github.com/apache/incubator-gluten/issues/8208, we could add the global source folders for finer grained code activation control for different Spark versions.
These folders could be:
1. src-spark32, src-spark33, ...
2. src-delta-spark32, src-delta-spark33, ...
3. src-iceberg-spark32, src-iceberg-spark33, ...
4. ...
## Related previous PRs
https://github.com/apache/incubator-gluten/pull/8800
https://github.com/apache/incubator-gluten/pull/8256
Contributor guide
Research direction
Start by comparing the Maven configurations in backends-velox/pom.xml and backends-clickhouse/pom.xml, then review issue 8208 and PRs 8800 and 8256 for the earlier approach. Done means the Maven modules can selectively activate source folders for different Spark and integration versions without mismatched compilation or IntelliJ IDEA errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100