[Bug] Docker image build script with Hadoop3.2 does not work with Netty and Epoll
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Building a docker image via `deploy/kubernetes/docker/build.sh --hadoop-version 3.2.4` and using that with Netty and Epoll fails because the image contains the Hadoop binaries, which contain a different `netty-all` jar then the RSS shuffle server.
This can be fixed by
- either not include the Hadoop binaries in the Docker image
- or by downgrading Netty version to match the version contained in the Hadoop binaries
### Affects Version(s)
master
### Uniffle Server Log Output
```logtalk
Exception in thread "main" java.lang.UnsatisfiedLinkError: failed to load the required native library
at io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)
at io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:57)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:189)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:37)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:60)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:49)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:117)
at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104)
at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:81)
at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:57)
at org.apache.uniffle.server.netty.StreamServer.(StreamServer.java:88)
at org.apache.uniffle.server.ShuffleServer.initialization(ShuffleServer.java:300)
at org.apache.uniffle.server.ShuffleServer.(ShuffleServer.java:113)
at org.apache.uniffle.server.ShuffleServer.main(ShuffleServer.java:131)
Caused by: java.lang.ExceptionInInitializerError
at io.netty.channel.epoll.Epoll.(Epoll.java:40)
... 15 more
Caused by: java.lang.IllegalStateException: Multiple resources found for 'META-INF/native/libnetty_transport_native_epoll_x86_64.so' with different content: [jar:file:/data/rssadmin/rss/jars/server/netty-transport-native-epoll-4.1.109.Final-linux-x86_64.jar!/META-INF/native/libnetty_transport_native_epoll_x86_64.so, jar:file:/data/rssadmin/hadoop/share/hadoop/hdfs/lib/netty-all-4.1.68.Final.jar!/META-INF/native/libnetty_transport_native_epoll_x86_64.so]
at io.netty.util.internal.NativeLibraryLoader.getResource(NativeLibraryLoader.java:301)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:334)
at io.netty.channel.epoll.Native.(Native.java:96)
... 16 more
```
```
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start with deploy/kubernetes/docker/build.sh and reproduce the Hadoop 3.2.4 image build with Netty and Epoll enabled. Inspect the Hadoop and RSS server jars involved in the logged native-library conflict; done means the image no longer provides conflicting Netty native resources and the shuffle server starts successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, hadoop, java
- Domain
- build-system, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100