lablup / lablup/backend.ai

Run model service shell command via a temp script with GitHub Actions parity; verify multiline SIGTERM/error behavior

Open
#12,393 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Background
With the model service command stored as a single string (BA-6550) and the default shell kept as /bin/bash, the bash path is the common path. Today kernel/service.py wraps a string command as [shell, '-c', command] (src/ai/backend/kernel/service.py:117-122), which (a) reintroduces the shell signal-forwarding problem for multiline scripts, (b) silently launches the server even when a setup line failed, and (c) is hard to debug. This story moves execution to a GitHub Actions-style temp script and empirically verifies the runtime behavior.

Scope
- kernel/service.py: when service.shell is set (default /bin/bash), write the resolved command string to a uniquely-named temp script via mkstemp (e.g. /tmp/bai-start-XXXXXX.sh, mode 0700) and run [shell, '-~~noprofile', '~~-norc', '-eo', 'pipefail', ]. The bash-specific flags apply only when the shell is bash; fall back to a safe subset (e.g. -e) or '<shell> <script>' for non-bash shells.
- When service.shell is explicitly null/empty: shlex.split the command and exec the argv directly (no shell, no temp script) for clean SIGTERM delivery.
- Keep legacy list-form command working (cmdargs = [\*command]) for backward compatibility.
- Temp script lifecycle: random name (never a fixed /tmp/start.sh), restrictive perms, and a cleanup decision (best-effort unlink vs leave-for-debug).
- Empirically verify in a real container (model-service e2e stack): (1) SIGTERM reaches the actual server process for single-command AND multiline scripts, or document+implement the 'exec <server>' / signal-forwarding requirement; (2) effect of -e / pipefail on a failing setup line; (3) whether /tmp is a forced tmpfs and is writable.

Depends on BA-6550 (string representation + default shell semantics).

Success Criteria
- [ ] shell set (default /bin/bash) + single-command string -> temp script created, run via bash --noprofile --norc -eo pipefail <script>, server starts
- [ ] shell set + multiline string -> lines run in order; a failing intermediate line aborts before launching the server (set -e)
- [ ] pipefail: a failing producer in a pipe (e.g. false | tee x) yields non-zero exit
- [ ] shell explicitly null -> command is shlex.split and exec'd as argv, no temp script, no sh -c
- [ ] legacy list-form command -> still launches (backward compat)
- [ ] temp script name is randomized (mkstemp); two concurrent services do not collide; a pre-existing /tmp/start.sh in the image is not used
- [ ] SIGTERM behavior verified: server process receives termination for single AND multiline (or exec/forwarding documented + implemented)
- [ ] /tmp writability / tmpfs assumption verified and documented
- [ ] pants test passes for affected packages

JIRA Issue: BA-6615

Contributor guide

Open the contributing guide

Research direction

Start in src/ai/backend/kernel/service.py:117-122 and review the BA-6550 dependency before changing command execution. Verify the behavior in the real model-service e2e stack, including SIGTERM, failing setup lines, pipefail, and /tmp writability; then run the affected packages' Pants tests. Done means all listed success criteria pass, including string, null-shell, and legacy list-form commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, docker, python
Domain
backend, devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.