lablup / lablup/backend.ai

Image environment variables colliding with injected krunner Python, causing kernel launcher startup failure

Open
#10,764 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Motivation

backend.ai decouples its kernel runner (krunner) from user-provided container images by mounting a statically built Python distribution at `/opt/backend.ai` and launching the kernel via `/opt/backend.ai/bin/python -m ai.backend.kernel`. This design allows any compatible container image to run as a [backend.ai](http://backend.ai) session without requiring image authors to pre-install Backend.AI-specific software.

However, certain container images — particularly ML/inference frameworks like vLLM — ship with their own Python virtual environments and bake environment variables such as `VIRTUAL_ENV` and modified `PATH` directly into the image's `ENV` directives. When backend.ai spawns a container from such an image, these pre-existing environment variables leak into the execution context of the krunner's statically built Python, causing fatal import failures before the kernel launcher can even start.

The issue is reliably reproducible with images like `intel-vllm:0.17.0-xpu`, where the image sets `VIRTUAL_ENV` pointing to its internal venv. When the krunner Python at `/opt/backend.ai/bin/python` inherits this `VIRTUAL_ENV`, CPython's startup sequence attempts to resolve its standard library modules (e.g., `encodings`) against the image's virtual environment `site-packages` rather than the krunner's own bundled standard library. This results in a fatal `ModuleNotFoundError: No module named 'encodings'` before any application code has a chance to run:

```
$ docker run --rm -it \
-v backendai-krunner.v14.x86_64.static-gnu:/opt/backend.ai \
-v .../ai/backend/kernel:/opt/backend.ai/lib/python3.13/site-packages/ai/backend/kernel \
/intel-vllm:0.17.0-xpu /opt/backend.ai/bin/python -m ai.backend.kernel
Could not find platform independent libraries
Could not find platform dependent libraries
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007efb7d995c40 (most recent call first):

```

Unsetting `VIRTUAL_ENV` at container launch immediately resolves the import failure, confirming the root cause:

```
$ docker run --rm -it \
-v backendai-krunner.v14.x86_64.static-gnu:/opt/backend.ai \
-v .../ai/backend/kernel:/opt/backend.ai/lib/python3.13/site-packages/ai/backend/kernel \
-e VIRTUAL_ENV \
/intel-vllm:0.17.0-xpu /opt/backend.ai/bin/python -m ai.backend.kernel
usage: __main__.py [-h] [--debug] {app,python,vendor.aws_polly,vendor.h2o} [runtime_path]
```

As backend.ai's core value proposition is the ability to bring any container image and run computational sessions seamlessly, this class of failure undermines that promise. Images from NVIDIA NGC, Intel's AI toolkit, Hugging Face, and various Conda-based distributions commonly set `VIRTUAL_ENV`, `PYTHONHOME`, `PYTHONPATH`, `CONDA_DEFAULT_ENV`, or prepend framework-specific directories to `PATH` — all of which can interfere with the krunner Python in similar ways. The agent should proactively sanitize or override these environment variables during container creation to ensure the krunner always boots correctly, regardless of the image's own environment configuration.

## Required Features

The agent's container creation logic (in the Docker backend under `src/ai/backend/agent/docker/`) should be updated to explicitly unset or override a defined set of Python-related environment variables that are known to interfere with the krunner's statically built Python. At minimum, the following variables should be cleared or overridden when constructing the container's environment:

- `VIRTUAL_ENV` — causes CPython to prepend the venv's `site-packages` to `sys.path` and alter `sys.prefix`, which breaks the krunner Python's module resolution. This is the confirmed root cause of the reported failure.
- `PYTHONHOME` — overrides CPython's `sys.prefix` and `sys.exec_prefix` entirely, redirecting standard library lookups away from the krunner's bundled stdlib at `/opt/backend.ai/lib/python3.X/`.
- `PYTHONPATH` — prepends arbitrary directories to `sys.path`, which can shadow the krunner's own packages or inject incompatible modules into the krunner's import chain.
- `CONDA_DEFAULT_ENV` and `CONDA_PREFIX` — Conda-based images use these to activate environments at shell startup, which can alter `PATH` and library loading in ways that conflict with the krunner.

The override should happen at the container environment construction phase (where the agent already sets environment variables like `LOCAL_USER_ID`, `LOCAL_GROUP_ID`, and core count variables from `ai.backend.envs.corecount`). These overrides should only apply to the krunner's execution context — the image's original environment should remain intact for user-facing processes running inside the session (e.g., Jupyter kernels, user scripts). This can be achieved by scoping the overrides to the entrypoint/command that launches the krunner, or by having the krunner's entrypoint script (`entrypoint.sh`) explicitly unset these variables before invoking `/opt/backend.ai/bin/python`.

The `PATH` variable requires special handling: rather than clearing it entirely (which would break user tools), the agent or entrypoint script should ensure that `/opt/backend.ai/bin` is positioned appropriately in `PATH` while preserving the image's original paths for user processes. Additionally, any future additions to the krunner's entrypoint script should include a well-documented block that sanitizes these variables with comments explaining why each override exists.

## Impact

- `src/ai/backend/agent/docker/` — the container creation and environment setup logic in the Docker backend, where environment variables are assembled before passing to the container runtime.
- `src/ai/backend/agent/` — the abstract agent layer that defines environment preparation interfaces used by all container backends (Docker, Kubernetes, etc.).
- Krunner entrypoint scripts — the `entrypoint.sh` or equivalent bootstrap script injected into containers, which is the last opportunity to sanitize the environment before invoking `/opt/backend.ai/bin/python -m ai.backend.kernel`.
- The `backend.ai-krunner-static-gnu` and `backend.ai-krunner-alpine` packages — if the fix is implemented at the krunner level rather than the agent level, the krunner's wrapper scripts or Python startup configuration (`site.py`, `_pth` files) may need to be updated.
- Documentation for image compatibility (`docs/dev/adding-kernels.rst`) — should note the environment variable sanitization behavior so that image authors understand that [backend.ai](http://backend.ai) will override certain Python-related variables during krunner initialization.

## Testing Scenarios

1. Launch a session using an image that sets `VIRTUAL_ENV` in its Dockerfile (e.g., a vLLM or NGC image) and verify that the krunner starts successfully without the `Failed to import encodings module` error. The kernel launcher should proceed to its normal argument parsing phase.
1. Launch a session using an image that sets `PYTHONHOME` to a custom path and verify that the krunner's statically built Python correctly resolves its own standard library modules and site-packages without being redirected by the image's `PYTHONHOME`.
1. Launch a session using a Conda-based image (where `CONDA_DEFAULT_ENV`, `CONDA_PREFIX`, and a modified `PATH` are baked into the image's environment) and verify that the krunner starts correctly while user processes within the session can still access the Conda environment's tools and packages.
1. After the krunner starts successfully, verify that user-facing processes (Jupyter, terminal sessions, user scripts) still have access to the image's original Python environment and installed packages. The environment sanitization must be scoped to krunner initialization only — clearing `VIRTUAL_ENV` globally would break user workflows that depend on the image's Python environment.
1. Launch a session using a standard Backend.AI-provided image (e.g., `lablup/python:3.X-ubuntu`) that does not set any conflicting environment variables, and verify that the fix introduces no regression — the krunner should start exactly as before.
1. Launch a session using an image that sets `PYTHONPATH` to include directories containing Python packages with the same names as krunner dependencies (e.g., a custom `zmq` or `msgpack` module), and verify that the krunner loads its own bundled versions rather than the image's conflicting packages.

JIRA Issue: BA-5582

Contributor guide

Open the contributing guide

Research direction

Trace environment assembly in src/ai/backend/agent/docker/ and the abstract agent layer, then inspect the krunner entrypoint.sh or equivalent bootstrap script. Reproduce the failure with an image setting VIRTUAL_ENV or PYTHONHOME, and verify that krunner starts while user-facing processes retain the image environment. Check docs/dev/adding-kernels.rst if the behavior is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.