lablup / lablup/backend.ai

Incorrect exit code returned when Backend.AI server component crashes

Open
#10,717 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Summary

All Backend.AI microservice components (manager, agent, storage, account-manager, web, appproxy-coordinator, appproxy-worker) always exit with code 0 regardless of whether the child worker process terminated normally or crashed with an unhandled exception. This makes it impossible for orchestrators like systemd to detect abnormal termination and take corrective action (restart, alerting).

## Steps to Reproduce

1. Start any Backend.AI service (e.g., `./backend.ai mgr start-server`)
1. Inject a fatal error during worker initialization (e.g., misconfigure the database connection to cause an unhandled
exception in server_main)
1. Observe the child worker crash with a traceback
1. Check the parent process exit code with `echo $?`

## Expected Behavior

- The parent process should exit with a non-zero exit code (e.g., 1) when the child worker terminated due to an unhandled
exception, so that orchestrators can detect the failure.

## Actual Behavior

- The parent process always exits with code 0. aiotools.start_server() collects child exit codes internally but returns None, and each service's server_main_logwrapper swallows exceptions in its except Exception: traceback.print_exc() block without signaling the parent.

JIRA Issue: BA-5530

Contributor guide

Open the contributing guide

Research direction

Start by reading aiotools.start_server() and the services' server_main_logwrapper exception-handling paths described in the issue. Reproduce the failure with ./backend.ai mgr start-server and a deliberately broken database configuration, then verify that an unhandled worker failure causes the parent process to return a non-zero exit code while normal shutdown remains successful.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.