Kernel runner Python cannot load on images that omit glibc's compatibility stub libraries (libutil.so.1, libdl.so.2, libpthread.so.0, librt.so.1)
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
The static-gnu kernel runner Python (`/opt/backend.ai/bin/python`) is built for manylinux2014 and therefore declares `libpthread.so.0`, `libdl.so.2`, `libutil.so.1` and `librt.so.1` as needed libraries, next to `libm.so.6` and `libc.so.6`. The krunner volume ships only `libpython`; the rest must come from the image. Since glibc 2.34 those four libraries are empty compatibility stubs that distro `libc6` packages still install, but trimmed rootfs images such as NVIDIA NIM drop them and keep only `libc.so.6` and `libm.so.6`. On such an image every use of the kernel runner Python fails immediately:
```
/opt/backend.ai/bin/python: error while loading shared libraries: libutil.so.1: cannot open shared object file: No such file or directory
```
so the session container exits before the kernel runner starts. Present on main, 26.8 and 26.4; independent of BA-7867/BA-7868/BA-7869.
Verified with the real krunner archive: on a glibc 2.41 image with the four files deleted the Python fails to load; with glibc 2.34+ stub files placed on `LD_LIBRARY_PATH` it runs fully (`openpty`, `dlopen`, clocks). The same stubs must not be used on glibc older than 2.34 (there the four libraries are real and the stubs break symbol versioning), so the fix has to be gated by the image's glibc version, which the agent already resolves as the image distro.
### Reproduction
Create a session on a label-less or labelled `ubuntu22.04`-class image whose rootfs lacks `/lib/-linux-gnu/libutil.so.1` (for example an NVIDIA NIM image); the container log shows the loader error above.
### Expected behavior
For images whose glibc is 2.34 or newer, the agent provides the four compatibility stubs (per architecture) inside the container and puts their directory on `LD_LIBRARY_PATH` in the container environment, so the kernel runner Python loads whether or not the image carries the stubs. Images with older glibc are unchanged.
JIRA Issue: BA-7881
Contributor guide
Research direction
Start by locating the agent's image-distro resolution and the code that prepares the kernel runner container environment, then reproduce the loader failure on an image missing the four libraries. Done means glibc 2.34+ images receive architecture-appropriate compatibility stubs through LD_LIBRARY_PATH, while older glibc images remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100