lablup / lablup/backend.ai

Kernel runner Python cannot load on images that omit glibc's compatibility stub libraries (libutil.so.1, libdl.so.2, libpthread.so.0, librt.so.1)

Open
#14,611 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

The static-gnu kernel runner Python (`/opt/backend.ai/bin/python`) is built for manylinux2014 and therefore declares `libpthread.so.0`, `libdl.so.2`, `libutil.so.1` and `librt.so.1` as needed libraries, next to `libm.so.6` and `libc.so.6`. The krunner volume ships only `libpython`; the rest must come from the image. Since glibc 2.34 those four libraries are empty compatibility stubs that distro `libc6` packages still install, but trimmed rootfs images such as NVIDIA NIM drop them and keep only `libc.so.6` and `libm.so.6`. On such an image every use of the kernel runner Python fails immediately:

```
/opt/backend.ai/bin/python: error while loading shared libraries: libutil.so.1: cannot open shared object file: No such file or directory
```

so the session container exits before the kernel runner starts. Present on main, 26.8 and 26.4; independent of BA-7867/BA-7868/BA-7869.

Verified with the real krunner archive: on a glibc 2.41 image with the four files deleted the Python fails to load; with glibc 2.34+ stub files placed on `LD_LIBRARY_PATH` it runs fully (`openpty`, `dlopen`, clocks). The same stubs must not be used on glibc older than 2.34 (there the four libraries are real and the stubs break symbol versioning), so the fix has to be gated by the image's glibc version, which the agent already resolves as the image distro.

### Reproduction

Create a session on a label-less or labelled `ubuntu22.04`-class image whose rootfs lacks `/lib/-linux-gnu/libutil.so.1` (for example an NVIDIA NIM image); the container log shows the loader error above.

### Expected behavior

For images whose glibc is 2.34 or newer, the agent provides the four compatibility stubs (per architecture) inside the container and puts their directory on `LD_LIBRARY_PATH` in the container environment, so the kernel runner Python loads whether or not the image carries the stubs. Images with older glibc are unchanged.

JIRA Issue: BA-7881

Contributor guide

Open the contributing guide

Research direction

Start by locating the agent's image-distro resolution and the code that prepares the kernel runner container environment, then reproduce the loader failure on an image missing the four libraries. Done means glibc 2.34+ images receive architecture-appropriate compatibility stubs through LD_LIBRARY_PATH, while older glibc images remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.