suggestion to improve spark shims - use annotation like Apache auron to match different spark versions
Open
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
You can see my PR that I did for Apache Comet:
- https://github.com/apache/datafusion-comet/pull/4657
the code will look much cleaner IMO
Contributor guide
Research direction
Start by reviewing the referenced Apache Comet PR 4657 and comparing its annotation-based approach with Iceberg's Spark shims. Identify the relevant Spark-version compatibility code in Iceberg, then confirm that the shims use the annotation approach and remain compatible across the supported Spark versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100