lablup / lablup/backend.ai

Enhance usability of runtime variant feature

Open
#3,561 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Motivation  

Model service’s Runtime Variant feature is a way to eliminate process of writing down model definition file when launching the model service. Currently, to launch the model service with the runtime variant feature enabled, the user first have to load the supported image to the [Backend.AI](http://Backend.AI), then start the model service with both valid image and runtime variant option selected. This process misleads most of the user without the context - they understand the dropdown lets them gain access to the selected inference framework even on unsupported images.

![image-20250203-053524.png](image-20250203-053524.png){width=671 height=222}

## Required Features

- Update [Backend.AI](http://Backend.AI) Image label specification to explicitly advertise which runtime variant the image supports
- Revamp both Manager API and Web UI to restrict user’s runtime variant selection
- But we still need user to allow making use of image’s arbitrary `CMD` field to start the inference framework
- Initial idea is to remove all other variants from the dropdown and leave only `Custom (Default)` and `Predefined Image Command`, and create a new text area (which is hidden by default) to expose the selected image’s supported runtime variant
- Add new `runtime_variant` field to image GQL
- Ideation - will a single image support more than one runtime variants?

## Impact  

- [Backend.AI](http://Backend.AI)’s service creation (`POST /service`) API will be updated
- [Backend.AI](http://Backend.AI) WebUI will be updated

## Testing Scenarios  

WIP

JIRA Issue: BA-628

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.