Run model service shell command via a temp script with GitHub Actions parity; verify multiline SIGTERM/error behavior
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Background
With the model service command stored as a single string (BA-6550) and the default shell kept as /bin/bash, the bash path is the common path. Today kernel/service.py wraps a string command as [shell, '-c', command] (src/ai/backend/kernel/service.py:117-122), which (a) reintroduces the shell signal-forwarding problem for multiline scripts, (b) silently launches the server even when a setup line failed, and (c) is hard to debug. This story moves execution to a GitHub Actions-style temp script and empirically verifies the runtime behavior.
Scope
- kernel/service.py: when service.shell is set (default /bin/bash), write the resolved command string to a uniquely-named temp script via mkstemp (e.g. /tmp/bai-start-XXXXXX.sh, mode 0700) and run [shell, '-~~noprofile', '~~-norc', '-eo', 'pipefail', ]. The bash-specific flags apply only when the shell is bash; fall back to a safe subset (e.g. -e) or '<shell> <script>' for non-bash shells.
- When service.shell is explicitly null/empty: shlex.split the command and exec the argv directly (no shell, no temp script) for clean SIGTERM delivery.
- Keep legacy list-form command working (cmdargs = [\*command]) for backward compatibility.
- Temp script lifecycle: random name (never a fixed /tmp/start.sh), restrictive perms, and a cleanup decision (best-effort unlink vs leave-for-debug).
- Empirically verify in a real container (model-service e2e stack): (1) SIGTERM reaches the actual server process for single-command AND multiline scripts, or document+implement the 'exec <server>' / signal-forwarding requirement; (2) effect of -e / pipefail on a failing setup line; (3) whether /tmp is a forced tmpfs and is writable.
Depends on BA-6550 (string representation + default shell semantics).
Success Criteria
- [ ] shell set (default /bin/bash) + single-command string -> temp script created, run via bash --noprofile --norc -eo pipefail <script>, server starts
- [ ] shell set + multiline string -> lines run in order; a failing intermediate line aborts before launching the server (set -e)
- [ ] pipefail: a failing producer in a pipe (e.g. false | tee x) yields non-zero exit
- [ ] shell explicitly null -> command is shlex.split and exec'd as argv, no temp script, no sh -c
- [ ] legacy list-form command -> still launches (backward compat)
- [ ] temp script name is randomized (mkstemp); two concurrent services do not collide; a pre-existing /tmp/start.sh in the image is not used
- [ ] SIGTERM behavior verified: server process receives termination for single AND multiline (or exec/forwarding documented + implemented)
- [ ] /tmp writability / tmpfs assumption verified and documented
- [ ] pants test passes for affected packages
JIRA Issue: BA-6615
Contributor guide
Research direction
Start in src/ai/backend/kernel/service.py:117-122 and review the BA-6550 dependency before changing command execution. Verify the behavior in the real model-service e2e stack, including SIGTERM, failing setup lines, pipefail, and /tmp writability; then run the affected packages' Pants tests. Done means all listed success criteria pass, including string, null-shell, and legacy list-form commands.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, docker, python
- Domain
- backend, devops, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100