lablup / lablup/backend.ai

Offload model service's health check process to AppProxy

Open
#3,052 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

### Main idea

- On current implementation, each model server's healthiness is checked by the kernel runner.
- By offloading the process to AppProxy, an external component, we can detect if the container itself (or even physical server) went down.
- AppProxy is designed to be ran on a High-Availability setup. To prevent duplicate status updates and overloading traffics, we need to make sure only one among every AppProxy components hold the key to make health check requests.

### Common Requirements

- When possible, try to follow existing code and naming convention, including variable, API and database column name.
- Focus on implementing core functionalities only and avoid generating excessive fancy features - like unit test, performance test and metric exporter.

### Integration Steps

- New endpoint sync API schema (`POST /endpoint/{endpoint_id`})
- Include health check information for given kernels
- Availability (health check feature is optional)
- Target HTTP path and method
- Expected status codes
- Timeout threshold for each request
- Request interval
- Max retry count before status transition
- Version newer schema as `v2`
- Treat request bodies omitting the schema identification as `v1` request
- Try to match existing `model-definition.yaml` file’s health check schema as possible:
- ```
t.Dict({
t.Key("interval", default=10): t.Null | t.ToFloat[0:],
t.Key("path"): t.String,
t.Key("max_retries", default=10): t.Null | t.ToInt[1:],
t.Key("max_wait_time", default=15): t.Null | t.ToFloat[0:],
t.Key("expected_status_code", default=200): t.Null | t.ToInt[100:],
})
```
- Write down alembic migration schema and ORM definitions
- Create new `endpoint` table to store model service specific datas
- `id` (GUID, Primary Key)
- endpoint ID (offloaded from `circuit.endpoint_id`)
- Health check availability (bool)
- Health check information (`StructuredJSONObjectColumn`)
- Health check attributes (retry counter, …)
- Create new `endpoint` table for `circuit`s with `endpoint_id` populated
- Update `circuit` table to make `endpoint_id` point `endpoint` table’s Primary Key
- Store received health check metadata on database
- Add new `bai_redis` config directive on Coordinator configuration
- Used lated to publish model service’s health check related event to [Backend.AI](http://Backend.AI) Core cluster
- Follow the configuration schema of `ServerConfig.redis` section
- Spawn a health checking timer task on coordinator
- Wrap the timer task with `GlobalTimer` so that the atomicity can be guaranteed to health checker
- From timer task, iterate through every model services with health check option enabled
- Determine health status for each model service based on the health check response and retry counts
- Fire `ai.backend.common.events.model_serving.ModelServiceStatusEvent` whenever model service’s health status has updated through `bai_redis`’ `REDIS_STREAM_DB`
- Add option on AppProxy coordinator configuration file to customize timer fire period

JIRA Issue: BA-62

Contributor guide

Open the contributing guide

Research direction

Start with the POST /endpoint/{endpoint_id} sync API and the existing model-definition.yaml health-check schema, then trace the endpoint and circuit ORM definitions and alembic migrations. Review Coordinator and AppProxy configuration, GlobalTimer, and ModelServiceStatusEvent usage; done means the v2 schema, persisted metadata, HA-safe health-check task, Redis event publication, and configurable timer are integrated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
api, backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.