Support lower Hadoop versions in client-mr
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
Currently, uniffle use the default Hadoop version of 2.8.5.
When using the `./build_distribution.sh --spark2-profile 'spark2' --spark3-mvn '-Dspark.version=2.4.3' --spark3-profile 'spark3' --spark3-mvn '-Dspark.version=3.1.1' -Dhadoop.version=2.6.0`, it will throw exceptions due to some methods and vars not supported in Hadoop 2.6.0.
Some non-compatible params and methods as follows
1. `CallContext`, introduced by >= 2.8.0.
2. `MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT` introduced by 2.8.0. [ticket link](https://issues.apache.org/jira/browse/MAPREDUCE-6105)
3. `MRApps.getSystemPropertiesToLog` introduced by 2.8.0 [ticket link](https://issues.apache.org/jira/browse/MAPREDUCE-5232)
I think we could use the reflection to be compatible with lower hadoop version.
Contributor guide
Research direction
Start with the client-mr code paths that reference CallContext, MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT, and MRApps.getSystemPropertiesToLog. Reproduce the reported failure with the build_distribution.sh command and Hadoop 2.6.0, then compare behavior with Hadoop 2.8.5. Done means the lower-version build no longer throws these compatibility exceptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100