lablup / lablup/backend.ai

Advanced High-Availability considerations for model service

Open
#3,051 1 comment 0 reactions 1 assignee Claimed by @kyujin-cho View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Currently Model Service relies on the health check information provided by the kernel runner operating on each container. As the container itself acts as the only source, the health status cannot be determined whenever entire GPU node shuts down.To guarantee the activeness of each model service, it is crucial to check whether the container itself is unresponsive and try to reconcile the replica size if it is. We can suggest following improvements to resolve the issue:

- Make AppProxy as the health checker
- Add an option to automatically terminate unhealthy sessions after a certain grace period

JIRA Issue: BA-63

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.