Incorrect exit code returned when Backend.AI server component crashes
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Summary
All Backend.AI microservice components (manager, agent, storage, account-manager, web, appproxy-coordinator, appproxy-worker) always exit with code 0 regardless of whether the child worker process terminated normally or crashed with an unhandled exception. This makes it impossible for orchestrators like systemd to detect abnormal termination and take corrective action (restart, alerting).
## Steps to Reproduce
1. Start any Backend.AI service (e.g., `./backend.ai mgr start-server`)
1. Inject a fatal error during worker initialization (e.g., misconfigure the database connection to cause an unhandled
exception in server_main)
1. Observe the child worker crash with a traceback
1. Check the parent process exit code with `echo $?`
## Expected Behavior
- The parent process should exit with a non-zero exit code (e.g., 1) when the child worker terminated due to an unhandled
exception, so that orchestrators can detect the failure.
## Actual Behavior
- The parent process always exits with code 0. aiotools.start_server() collects child exit codes internally but returns None, and each service's server_main_logwrapper swallows exceptions in its except Exception: traceback.print_exc() block without signaling the parent.
JIRA Issue: BA-5530
Contributor guide
Research direction
Start by reading aiotools.start_server() and the services' server_main_logwrapper exception-handling paths described in the issue. Reproduce the failure with ./backend.ai mgr start-server and a deliberately broken database configuration, then verify that an unhandled worker failure causes the parent process to return a non-zero exit code while normal shutdown remains successful.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100