Restarted sessions are broken
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
Once a session is restarted, Backend.AI performs the followings:
- In the Agent:\* Destroys the container.
- Recreates the container reusing the prior allocation map and kernel resource spec.
- In the Manager:\* Just updates the kernel/session status (RUNNING → RESTARTING → RUNNING).
Though, the restarted session is broken:
- Connecting to in-container apps does not work, via both AppProxy and manager's own proxy like `./backend.ai app` command.\* Probably because the port mapping is changed but not propagated to Manager (and App Proxy).
- GPU config env-vars (#3253) have empty values.
- The `CID` field in `/home/config/resource.txt` gets duplicated after restarted. (#3266)
- maybe more?
Due to these issues, we need to restart the model service process in #3282, instead of cleanly restarting the container.
For future work, we may need to consider:
- Adding an option to rebuild the device allocation map, to avoid re-allocating "faulty" devices if present.
- Adding an explicit handling of restart failures in case of there is no available devices in such cases.
- #3284
JIRA Issue: BA-6
Contributor guide
Assessment
This issue has not been assessed yet.