Bundle `sudo` as intrinsic binary in backend.ai krunner
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
# Bundle self-contained `sudo` binary so we can ship `sudo` support to images without the binary
## Motivation
[http://Backend.AI](http://Backend.AI) 's agent already decouples runtime tooling from container images by mounting self-contained binaries into `/opt/kernel` at session startup. Utilities like `su-exec`, `tmux`, `sftp-server`, and `scp` are statically built and shipped as part of the krunner package (via `backend.ai-krunner-static-gnu` and `backend.ai-krunner-alpine`), so image authors never need to worry about bundling them. This is documented in the kernel image guide: the agent "uses `/opt/kernel` as the directory for mounting other self-contained single-binary utilities."
However, `sudo` itself is not currently part of this injected toolset. Many container images — especially minimal base images, customer-provided custom images, and stripped-down ML framework images — do not include the `sudo` binary. When a session user needs to install an extra OS-level package, start a privileged service, or perform any administrative task within the container, the absence of `sudo` forces them into workarounds: rebuilding the image with `sudo` pre-installed, asking an admin to do it, or using `su-exec` in ways it was not designed for. This is a recurring pain point that undermines the "bring any image" promise of the platform.
By bundling a self-contained `sudo` binary and injecting it the same way `su-exec` and `tmux` are already injected, [http://Backend.AI](http://Backend.AI) can provide seamless `sudo` support to every session regardless of what the underlying image ships. The sudoers configuration is already handled by the agent at session startup, so the only missing piece is the binary itself.
## Required Features
- A statically built (or otherwise fully self-contained) `sudo` binary must be produced for each supported platform combination: glibc-based Linux on x86_64 and aarch64, and musl-based Linux (Alpine) on x86_64 and aarch64. The binary should carry no dynamic dependencies on the container image's libraries so it works on any image meeting the manylinux2014 or musl 1.2 baseline.
- The build should be added to the existing krunner build pipelines in `backend.ai-krunner-static-gnu` and `backend.ai-krunner-alpine`, following the same Dockerfile and cross-compilation patterns already used for `ttyd`, `tmux`, and `su-exec`. The resulting binary should be included in the krunner wheel's data payload so it is extracted alongside the other `/opt/kernel` binaries when the agent deploys volumes.
- The agent's container creation logic (in `src/ai/backend/agent/docker/`) must be updated to mount the `sudo` binary into `/opt/kernel/sudo`. The binary must be owned by root and have the setuid bit set for `sudo` to function; this needs to be handled during volume preparation since the agent runs as root when setting up bind mounts. The sudoers configuration itself does not need new work — [http://Backend.AI](http://Backend.AI) already injects a sudoers file into containers by design, so the injected `sudo` binary can rely on the existing sudoers policy as-is.
- Because traditional `sudo` (written in C) has a deep dependency on PAM and other shared libraries, a Rust-based implementation such as `sudo-rs` from the Trifecta Tech Foundation should be evaluated as the primary candidate. `sudo-rs` is a memory-safe reimplementation that is being adopted by Ubuntu 25.10 as the default and has passed two independent security audits. If `sudo-rs` cannot be statically linked cleanly for all target triples, an alternative like a minimal C-based `sudo` built against musl (for static linking) or a purpose-built lightweight privilege escalation wrapper should be investigated.
- Rather than gating this behind a feature flag, the injected binary directory (e.g., `/opt/kernel`) should be appended to the very end of the container's `PATH` rather than prepended. This way, if an image already ships its own `sudo` binary in a standard location like `/usr/bin/sudo`, the image-native binary will naturally take precedence during lookup. The injected one only serves as a fallback for images that lack `sudo` entirely. This is a simple, zero-configuration approach that respects the image author's intent without requiring any label or per-image opt-in/opt-out.
## Impact
- `backend.ai-krunner-static-gnu` — new build target and binary artifact for the glibc-based sudo binary.
- `backend.ai-krunner-alpine` — new build target and binary artifact for the musl-based sudo binary.
- `src/ai/backend/agent/docker/` — container creation and volume mount logic to inject the sudo binary and set permissions (setuid, root ownership). The `/opt/kernel` directory (or a dedicated subdirectory for fallback binaries) must be appended to the end of the container's `PATH` environment variable so that image-native binaries always take precedence.
- `src/ai/backend/agent/kernel.py` (or equivalent entrypoint setup) — ensure the PATH ordering places the injected binary directory last.
- `docs/dev/adding-kernels.rst` — document that `sudo` is now available as a fallback in all sessions via `/opt/kernel`, and explain the PATH-last ordering so image authors understand that their own `sudo` binary always takes precedence.
- CI/CD workflows in the krunner repositories — extended build matrix to compile and test the sudo binary across all four platform targets (x86_64/aarch64 × glibc/musl).
## Testing Scenarios
1. Launch a session using a minimal base image (e.g., `ubuntu:24.04` or `alpine:3.21`) that does not have `sudo` installed. Verify that `sudo whoami` returns `root` and that `sudo apt-get update` (or `sudo apk update`) completes successfully without password prompts.
1. Launch a session using an image that already ships its own `sudo` binary (e.g., a full Ubuntu desktop image). Run `which sudo` and confirm it resolves to the image-native path (e.g., `/usr/bin/sudo`) rather than `/opt/kernel/sudo`, since the injected directory sits at the end of `PATH`. Verify that the image's native `sudo` operates normally.
1. On an image that does not ship `sudo`, verify that `which sudo` resolves to the injected fallback path (e.g., `/opt/kernel/sudo`) and that it functions correctly.
1. Run sessions on both x86_64 and aarch64 agent nodes, and on both glibc-based (Ubuntu/CentOS/Debian) and musl-based (Alpine) images, to confirm the correct platform-specific binary is injected in each case.
1. Verify that the injected `sudo` binary has the correct ownership (`root:root`) and permissions (setuid bit set, mode `4755`) inside the container, and that a non-root session user can escalate privileges through it.
1. Confirm that multi-node cluster sessions (if applicable) receive the `sudo` binary on every kernel container, not only the main session container.
1. Attempt to use `sudo` to perform a destructive operation (e.g., `sudo rm -rf /opt/backend.ai`) and verify that [http://Backend.AI](http://Backend.AI) 's existing security mechanisms (jail, seccomp profiles, read-only mounts) appropriately restrict what the escalated user can actually do, so that `sudo` does not become an escape hatch for circumventing platform-level protections.
JIRA Issue: BA-4342
Contributor guide
Assessment
This issue has not been assessed yet.