lablup / lablup/backend.ai

Restart model service process without reloading the container

Open
#3,376 0 comments 0 reactions 1 assignee Claimed by @kyujin-cho View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Motivation  

In early bootstrap phases, model service needs to be restarted frequently to find out the working and performant parameter. In such cases following the original model service restart - terminating and creating a fresh new session - would be a bottleneck, especially when compared to just restarting the inference framework process itself. By extending current kernel runner spec a little bit mode, we can think achieve restarting the model service process itself while not touching the container lifecycle.

## Required Features

- Create a new REST API to trigger restarting the model service process
- Create a new RPC Handler to handle model service process restart request
- Add a new kernel runner interface which shuts down ongoing model service process

## Impact  

- [Backend.AI](http://Backend.AI) Agent: new RPC interface
- [Backend.AI](http://Backend.AI) Manager: new REST API

## Testing Scenarios  

- Trigger restart of the model service process and check if the inference worker loads like before
- Reiterate step 1, this time modifying the `model-definition.yaml` file. Check if the change is reflected to the runner.

JIRA Issue: BA-441

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.