bytedance / bytedance/CloudShuffleService
support Spark3.2 and Java11
- Dominant language
- Java
- Stars
- 260
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
### What would you like to be added?
I am trying to build a CSS docker image to support Spark3.2 based on java11 (FROM amazoncorretto:11 ), not able to compile it so far. How can I make it work with Spark3.2? It is acceptable if CSS works with Spark3.2 & java8. Please help.
Here is the list of version changes in pom.xml file:
```
11
${java.version}
${java.version}
3.8.6
3.2.0
3.2.1
```
Error:
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project css-common_2.12: Compilation failure: Compilation failure:
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[27,16] cannot find symbol
[ERROR] symbol: class Cleaner
[ERROR] location: package sun.misc
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[174,7] cannot find symbol
[ERROR] symbol: class Cleaner
[ERROR] location: class com.bytedance.css.common.unsafe.Platform
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[174,25] cannot find symbol
[ERROR] symbol: variable Cleaner
[ERROR] location: class com.bytedance.css.common.unsafe.Platform
[ERROR] -> [Help 1]
```
The replication step is:
1. spin up a jdk11 container with maven installed:
```
docker build -t jdk11 .
docker run -it jdk11
```
The jdk11 Dockerfile looks like this:
```
FROM amazoncorretto:11
ARG MAVEN_VERSION=3.8.6
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
# install maven
RUN yum update -y && yum install -y git tar
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
```
2. After login to the jdk11 container, upgrade spark & java versions in `pom.xml`, then compile the project:
```
git clone https://github.com/bytedance/CloudShuffleService.git /tmp/CloudShuffleService
cd /tmp/CloudShuffleService
vi pom.xml
./build.sh
```
### Why is this needed?
need to run the latest Spark with the CSS.
Contributor guide
Research direction
Start with pom.xml and common/src/main/java/com/bytedance/css/common/unsafe/Platform.java, then reproduce the reported failure using the provided Amazon Corretto 11 Dockerfile and ./build.sh. Determine the compatibility changes needed for Spark 3.2 with Java 11, or Java 8 if that is the supported outcome. Done means the project compiles successfully under the requested version combination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, spark
- Domain
- build-system, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100