bytedance / bytedance/CloudShuffleService

support Spark3.2 and Java11

Open
#4 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
260
Forks
57
PR merge metrics
No merged PRs in 30d

Description

### What would you like to be added?

I am trying to build a CSS docker image to support Spark3.2 based on java11 (FROM amazoncorretto:11 ), not able to compile it so far. How can I make it work with Spark3.2? It is acceptable if CSS works with Spark3.2 & java8. Please help.

Here is the list of version changes in pom.xml file:
```
11
${java.version}
${java.version}
3.8.6
3.2.0
3.2.1
```
Error:
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project css-common_2.12: Compilation failure: Compilation failure:
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[27,16] cannot find symbol
[ERROR] symbol: class Cleaner
[ERROR] location: package sun.misc
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[174,7] cannot find symbol
[ERROR] symbol: class Cleaner
[ERROR] location: class com.bytedance.css.common.unsafe.Platform
[ERROR] /tmp/CloudShuffleService/common/src/main/java/com/bytedance/css/common/unsafe/Platform.java:[174,25] cannot find symbol
[ERROR] symbol: variable Cleaner
[ERROR] location: class com.bytedance.css.common.unsafe.Platform
[ERROR] -> [Help 1]
```

The replication step is:
1. spin up a jdk11 container with maven installed:
```
docker build -t jdk11 .
docker run -it jdk11
```
The jdk11 Dockerfile looks like this:
```
FROM amazoncorretto:11
ARG MAVEN_VERSION=3.8.6
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries

# install maven
RUN yum update -y && yum install -y git tar
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
```
2. After login to the jdk11 container, upgrade spark & java versions in `pom.xml`, then compile the project:
```
git clone https://github.com/bytedance/CloudShuffleService.git /tmp/CloudShuffleService
cd /tmp/CloudShuffleService
vi pom.xml
./build.sh
```

### Why is this needed?

need to run the latest Spark with the CSS.

Contributor guide

Open the contributing guide

Research direction

Start with pom.xml and common/src/main/java/com/bytedance/css/common/unsafe/Platform.java, then reproduce the reported failure using the provided Amazon Corretto 11 Dockerfile and ./build.sh. Determine the compatibility changes needed for Spark 3.2 with Java 11, or Java 8 if that is the supported outcome. Done means the project compiles successfully under the requested version combination.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, spark
Domain
build-system, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.