apache / apache/uniffle

Support lower Hadoop versions in client-mr

Open
#160 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

Currently, uniffle use the default Hadoop version of 2.8.5.

When using the `./build_distribution.sh --spark2-profile 'spark2' --spark3-mvn '-Dspark.version=2.4.3' --spark3-profile 'spark3' --spark3-mvn '-Dspark.version=3.1.1' -Dhadoop.version=2.6.0`, it will throw exceptions due to some methods and vars not supported in Hadoop 2.6.0.

Some non-compatible params and methods as follows
1. `CallContext`, introduced by >= 2.8.0.
2. `MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT` introduced by 2.8.0. [ticket link](https://issues.apache.org/jira/browse/MAPREDUCE-6105)
3. `MRApps.getSystemPropertiesToLog` introduced by 2.8.0 [ticket link](https://issues.apache.org/jira/browse/MAPREDUCE-5232)

I think we could use the reflection to be compatible with lower hadoop version.

Contributor guide

Open the contributing guide

Research direction

Start with the client-mr code paths that reference CallContext, MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT, and MRApps.getSystemPropertiesToLog. Reproduce the reported failure with the build_distribution.sh command and Hadoop 2.6.0, then compare behavior with Hadoop 2.8.5. Done means the lower-version build no longer throws these compatibility exceptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.