gchq / gchq/sleeper

Rust build compatible with Amazon Linux

Open
#6,840 0 comments 0 reactions 1 assignee Claimed by @patchwork01 View on GitHub
build-pipeline enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a analytic writer, I want to run Spark jobs that can use Sleeper as a data source, so that I can perform complex Spark analytics against Sleeper without first having to export the required data as a separate preparation step. I would like to run this on Amazon EMR in a cluster with Spark configured.

As a user, I want Sleeper's DataFusion code to work on Amazon Linux, so that I can make direct queries against Sleeper from a machine running in Amazon Linux.

### Description / Background

Sleeper currently has a Spark integration capability. This is dependent on our Apache DataFusion code in Rust in order
to transfer Sleeper data as blocks of Arrow records straight to Spark.

The most recent EMR platform is based on Amazon Linux 2023. Unfortunately, the Rust code cannot be loaded on EMR Spark due to a version conflict in the required version of glibc (fundamental C runtime library/OS kernel interface).

This also applies to running any of our DataFusion code on any machine that runs Amazon Linux.

We'd like to make sure that any machine running Amazon Linux can run our DataFusion code.

There's also a problem with the current Rust builder images that they use the latest stable Rust toolchain, instead of the one we test with in CI.

### Acceptance Criteria

**When** I try to run an EMR Spark job using Sleeper
**Then** the DataFusion code library successfully loads and links.

### Technical Notes / Implementation Details

This issue should be considered in light of #6631.

#### Build on Amazon Linux

We've created a branch [al2023-build](https://github.com/gchq/sleeper/tree/al2023-build). This includes extra scripts to build the Rust code on Amazon Linux 2023. To test this:

1. `./rust/builders/buildAmazonLinux2023.sh` creates the Docker images locally
2. `SKIP_DOCKER_PULL=true ./scripts/build/build.sh` should trigger a build. The SKIP_DOCKER_PULL is only needed because obviously the image isn't published anywhere. If this approach was merged, you wouldn't need this.
3. Verify the GLIBC version `readelf -V path/to/libsleeper_df.so | sed -n 's/.*Name: \(GLIBC_[0-9.]*\).*/\1/p' | sort -Vu | tail -1` should be no higher than GLIBC2.34 (Amazon Linux 2023's version).

These native libs should now work on Amazon Linux 2023 in queries, Trino/Spark/EMR wherever else that annoying GLIBC error was appearing.

To get this to PR status:

1. Add sccache integration. Just needs some tweaks moving over from existing buildAllSccache.sh script.
2. Add certs handling.
3. Add a hook so the URL for the rustup.sh script can be directed to an offline location and it needs to be configurable.
4. Properly integrate the scripts, by replacing the existing build scripts and deleting the old docker files.
5. Add a rust-toolchain file and make that authoritative for the version (probably split this out as a separate issue).

#### Security

*Down*grading a build image OS should not introduce *runtime* vulnerabilities. It only affects the version of glibc that the native library requires to run. It does __not__ mean the code will actually run using the older glibc version. Other than EMR, the Sleeper compaction/query code can (and should) still be run in a up to date OS version, with an up to date glibc version. Therefore any bug fixes/security vulnerabilities fixes since this older version of glibc will still be present at runtime.

In addition the rest of the toolchain, e.g. Rust toolchain version, C/C++ compiler toolchain, Rust code dependencies will still use the most up to date versions.

#### Performance impact & testing

It's possible this may have an impact on performance of our Rust code, as it affects how we compile the DataSketches C++ code which runs against every row during a compaction. We can check performance with CompactionPerformanceST to make sure there's no significant impact.

#### Alternative with Debian Bullseye (decided against)

The Rust code is built inside a Docker container based on Debian 12 (codenamed Bookworm). From (https://packages.debian.org/search?searchon=sourcenames&keywords=glibc) we see that Bookworm ships with glibc 2.36.
Amazon Linux 2023 ships with glibc 2.34 (https://docs.aws.amazon.com/linux/al2023/ug/core-glibc.html). Thus, when EMR on AL2023 tries to load the libsleeper_df.so library, it fails due to the system glibc being out of date.

Looking at (https://stackoverflow.com/questions/76927676/how-can-i-build-a-program-to-run-on-a-system-with-older-libc) we see that attempting to install multiple versions of glibc or up/downgrading the existing glibc is a futile effort due to how intricately the system glibc library is embedded in the OS. The most commonly given answer is to build the code on an OS which is based off a sufficient old version of glibc.

From (https://packages.debian.org/search?searchon=sourcenames&keywords=glibc) we can see that Debian 11 (Bullseye) ships with glibc 2.31 which is old enough to run on AL2023. Also, if we look at (https://hub.docker.com/_/rust/tags?name=1.93) we can see that a Rust Docker image is available for Rust version 1.93 based on Debian Bullseye.

Therefore, if we downgrade the Rust Docker build images in rust/builders/base/Dockerfile to `rust-1.93:bullseye` we will create a native libsleeper_df.so that links against a glibc usable by Amazon EMR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.