lablup / lablup/backend.ai

Restarted sessions are broken

Open
#3,283 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Once a session is restarted, Backend.AI performs the followings:

- In the Agent:\* Destroys the container.
- Recreates the container reusing the prior allocation map and kernel resource spec.

- In the Manager:\* Just updates the kernel/session status (RUNNING → RESTARTING → RUNNING).

Though, the restarted session is broken:

- Connecting to in-container apps does not work, via both AppProxy and manager's own proxy like `./backend.ai app` command.\* Probably because the port mapping is changed but not propagated to Manager (and App Proxy).

- GPU config env-vars (#3253) have empty values.
- The `CID` field in `/home/config/resource.txt` gets duplicated after restarted. (#3266)
- maybe more?
Due to these issues, we need to restart the model service process in #3282, instead of cleanly restarting the container.

For future work, we may need to consider:

- Adding an option to rebuild the device allocation map, to avoid re-allocating "faulty" devices if present.
- Adding an explicit handling of restart failures in case of there is no available devices in such cases.
- #3284

JIRA Issue: BA-6

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.