[VL] Build: Dynamic build vs static build
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
We have put a lot of daily effort for maintaining the vcpkg-based static build. We can revisit the whole design to see if it's on the right direction.
Maintaining the static build is expensive: we should make sure all the needed libraries have the static versions of releases, and they are even better to be provided by vcpkg. Even one single unwanted dynamic dependency could break all the portability (e.g., libcudf), and we don't have much things to do with it except for manually wrapping it to the fat jar and loading it in code. AFAIK static linking also doesn't work very well for platforms other than x86 now, although it's eventually solvable but still needs human effort.
The initial reason why we promoted the static build way was for portability. However, dynamic linking could be just as effective: we can still put all the required dynamic libraries into the same Jar and make Java code load them all, so everything will work fine.
Although we are also maintaining the [dynamic loaders](https://github.com/apache/incubator-gluten/tree/main/backends-velox/src/main/scala/org/apache/gluten/utils), the design is a bit too complicated either. Loaders load the libraries one-by-one with a certain order (otherwise, crashing), and the portability is also questioned given the loaders are designed for certain Linux distributions.
For improving this part of code, [Velox4J](https://github.com/velox4j/velox4j)'s approach can be used as reference. It includes:
1. A CMake installation script to install all the runtime dependencies to installation folder, with all the system dependencies (e.g., libc.so) eliminated:
https://github.com/velox4j/velox4j/blob/4ed9f1a69e7c54c817c1057b17df720334314ede/src/main/cpp/main/CMakeLists.txt#L43-L71
2. A bash script to modify runpaths to `$ORIGIN` for libraries in the installation folder, to make sure the libraries find each other when being loaded:
https://github.com/velox4j/velox4j/blob/4ed9f1a69e7c54c817c1057b17df720334314ede/src/main/cpp/build.sh#L19-L61
3. A Java library loader implementation to copy all the libraries in the fat jar to a tmp loader and only load the `libvelox.so` (or whatever we want to load) itself, then all the required dependency libraries will be found from the tmp folder and be loaded automatically:
https://github.com/velox4j/velox4j/blob/4ed9f1a69e7c54c817c1057b17df720334314ede/src/main/java/io/github/zhztheplayer/velox4j/jni/JniLibLoader.java#L32-L73
Ideally that's all. By doing this we can drop VCPKG completely, the build process can also be simplified a lot. The libraries that are not with static version provided can be naturally portable by this approach, and the extra effort for supporting non-x86 platform in the static build can be avoided either.
Overall it's just a path-finding so far because the current build system is a bit fragile for such a large-scale refactoring.
Contributor guide
Assessment
This issue has not been assessed yet.