Restart model service process without reloading the container
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Motivation
In early bootstrap phases, model service needs to be restarted frequently to find out the working and performant parameter. In such cases following the original model service restart - terminating and creating a fresh new session - would be a bottleneck, especially when compared to just restarting the inference framework process itself. By extending current kernel runner spec a little bit mode, we can think achieve restarting the model service process itself while not touching the container lifecycle.
## Required Features
- Create a new REST API to trigger restarting the model service process
- Create a new RPC Handler to handle model service process restart request
- Add a new kernel runner interface which shuts down ongoing model service process
## Impact
- [Backend.AI](http://Backend.AI) Agent: new RPC interface
- [Backend.AI](http://Backend.AI) Manager: new REST API
## Testing Scenarios
- Trigger restart of the model service process and check if the inference worker loads like before
- Reiterate step 1, this time modifying the `model-definition.yaml` file. Check if the change is reflected to the runner.
JIRA Issue: BA-441
Contributor guide
Assessment
This issue has not been assessed yet.