Add scaling group options for health-based session termination
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Problem Statement
Scaling groups need configurable options to control health-based session termination policies. Different use cases require different health check strategies (readiness-only, liveness-only, or both).
## Background
- Scaling groups have `scheduler_opts` field (JSONB) for scheduler-specific configuration
- Currently, no health check termination policies exist
- Different deployment scenarios need different termination strategies:
- Model serving: May prioritize readiness for traffic routing
- Batch jobs: May only care about liveness
- Interactive sessions: May need both checks
## Implementation Tasks
### 1. Define Health Check Policy Schema
Add new options to `scheduler_opts` in `ScalingGroupRow`:
```python
{
"health_check_termination": {
"enabled": bool, # Enable/disable health-based termination
"check_readiness": bool, # Consider readiness failures
"check_liveness": bool, # Consider liveness failures
"grace_period_seconds": int, # Seconds to wait before termination
"max_consecutive_failures": int, # Consecutive failures before termination
"check_interval_seconds": int # How often to check health status
}
}
```
### 2. Add GraphQL API Support
- Update GraphQL schema to expose health check options
- Add mutation for updating scaling group health check policy
- Add query for retrieving current policy
- Validate options on input
### 3. Database Migration (if needed)
- No schema change required (using existing JSONB field)
- May need to add default values for existing scaling groups
- Document the new options in scaling group model
### 4. Add Validation Logic
- Validate that at least one of `check_readiness` or `check_liveness` is enabled if termination is enabled
- Validate reasonable ranges for grace period and check intervals
- Ensure backwards compatibility with existing scaling groups
### 5. Documentation
- Update API documentation for scaling group options
- Add examples of common configurations
- Document interaction with health check infrastructure
### 6. Testing
- Unit tests for option validation
- Integration tests with GraphQL API
- Test backwards compatibility with existing scaling groups
- Verify options are correctly used by Manager termination logic
## Acceptance Criteria
- [ ] Health check termination options can be configured per scaling group
- [ ] GraphQL API supports CRUD operations for health check options
- [ ] Default values work correctly for existing scaling groups
- [ ] Validation prevents invalid configurations
- [ ] Documentation is complete and clear
- [ ] Tests verify all functionality
## Example Configurations
### Model Serving (Readiness-focused)
```json
{
"health_check_termination": {
"enabled": true,
"check_readiness": true,
"check_liveness": false,
"grace_period_seconds": 60,
"max_consecutive_failures": 3,
"check_interval_seconds": 30
}
}
```
### Batch Processing (Liveness-focused)
```json
{
"health_check_termination": {
"enabled": true,
"check_readiness": false,
"check_liveness": true,
"grace_period_seconds": 300,
"max_consecutive_failures": 5,
"check_interval_seconds": 60
}
}
```
### Interactive Sessions (Both checks)
```json
{
"health_check_termination": {
"enabled": true,
"check_readiness": true,
"check_liveness": true,
"grace_period_seconds": 120,
"max_consecutive_failures": 3,
"check_interval_seconds": 30
}
}
```
## Reference Code
- `src/ai/backend/manager/models/scaling_group.py` - ScalingGroupRow model
- `src/ai/backend/manager/api/gql/scaling_group.py` - GraphQL API (if exists)
## Dependencies
- This story should be implemented before or in parallel with BA-3020 (Manager termination logic)
- Manager termination logic will consume these options
JIRA Issue: BA-3021
Contributor guide
Assessment
This issue has not been assessed yet.