Offload model service's health check process to AppProxy
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
### Main idea
- On current implementation, each model server's healthiness is checked by the kernel runner.
- By offloading the process to AppProxy, an external component, we can detect if the container itself (or even physical server) went down.
- AppProxy is designed to be ran on a High-Availability setup. To prevent duplicate status updates and overloading traffics, we need to make sure only one among every AppProxy components hold the key to make health check requests.
### Common Requirements
- When possible, try to follow existing code and naming convention, including variable, API and database column name.
- Focus on implementing core functionalities only and avoid generating excessive fancy features - like unit test, performance test and metric exporter.
### Integration Steps
- New endpoint sync API schema (`POST /endpoint/{endpoint_id`})
- Include health check information for given kernels
- Availability (health check feature is optional)
- Target HTTP path and method
- Expected status codes
- Timeout threshold for each request
- Request interval
- Max retry count before status transition
- Version newer schema as `v2`
- Treat request bodies omitting the schema identification as `v1` request
- Try to match existing `model-definition.yaml` file’s health check schema as possible:
- ```
t.Dict({
t.Key("interval", default=10): t.Null | t.ToFloat[0:],
t.Key("path"): t.String,
t.Key("max_retries", default=10): t.Null | t.ToInt[1:],
t.Key("max_wait_time", default=15): t.Null | t.ToFloat[0:],
t.Key("expected_status_code", default=200): t.Null | t.ToInt[100:],
})
```
- Write down alembic migration schema and ORM definitions
- Create new `endpoint` table to store model service specific datas
- `id` (GUID, Primary Key)
- endpoint ID (offloaded from `circuit.endpoint_id`)
- Health check availability (bool)
- Health check information (`StructuredJSONObjectColumn`)
- Health check attributes (retry counter, …)
- Create new `endpoint` table for `circuit`s with `endpoint_id` populated
- Update `circuit` table to make `endpoint_id` point `endpoint` table’s Primary Key
- Store received health check metadata on database
- Add new `bai_redis` config directive on Coordinator configuration
- Used lated to publish model service’s health check related event to [Backend.AI](http://Backend.AI) Core cluster
- Follow the configuration schema of `ServerConfig.redis` section
- Spawn a health checking timer task on coordinator
- Wrap the timer task with `GlobalTimer` so that the atomicity can be guaranteed to health checker
- From timer task, iterate through every model services with health check option enabled
- Determine health status for each model service based on the health check response and retry counts
- Fire `ai.backend.common.events.model_serving.ModelServiceStatusEvent` whenever model service’s health status has updated through `bai_redis`’ `REDIS_STREAM_DB`
- Add option on AppProxy coordinator configuration file to customize timer fire period
JIRA Issue: BA-62
Contributor guide
Research direction
Start with the POST /endpoint/{endpoint_id} sync API and the existing model-definition.yaml health-check schema, then trace the endpoint and circuit ORM definitions and alembic migrations. Review Coordinator and AppProxy configuration, GlobalTimer, and ModelServiceStatusEvent usage; done means the v2 schema, persisted metadata, HA-safe health-check task, Redis event publication, and configurable timer are integrated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- api, backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100