lablup / lablup/backend.ai

Add scaling group options for health-based session termination

Open
#6,718 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Problem Statement

Scaling groups need configurable options to control health-based session termination policies. Different use cases require different health check strategies (readiness-only, liveness-only, or both).

## Background

- Scaling groups have `scheduler_opts` field (JSONB) for scheduler-specific configuration
- Currently, no health check termination policies exist
- Different deployment scenarios need different termination strategies:
- Model serving: May prioritize readiness for traffic routing
- Batch jobs: May only care about liveness
- Interactive sessions: May need both checks

## Implementation Tasks

### 1. Define Health Check Policy Schema

Add new options to `scheduler_opts` in `ScalingGroupRow`:

```python
{
"health_check_termination": {
"enabled": bool, # Enable/disable health-based termination
"check_readiness": bool, # Consider readiness failures
"check_liveness": bool, # Consider liveness failures
"grace_period_seconds": int, # Seconds to wait before termination
"max_consecutive_failures": int, # Consecutive failures before termination
"check_interval_seconds": int # How often to check health status
}
}
```

### 2. Add GraphQL API Support

- Update GraphQL schema to expose health check options
- Add mutation for updating scaling group health check policy
- Add query for retrieving current policy
- Validate options on input

### 3. Database Migration (if needed)

- No schema change required (using existing JSONB field)
- May need to add default values for existing scaling groups
- Document the new options in scaling group model

### 4. Add Validation Logic

- Validate that at least one of `check_readiness` or `check_liveness` is enabled if termination is enabled
- Validate reasonable ranges for grace period and check intervals
- Ensure backwards compatibility with existing scaling groups

### 5. Documentation

- Update API documentation for scaling group options
- Add examples of common configurations
- Document interaction with health check infrastructure

### 6. Testing

- Unit tests for option validation
- Integration tests with GraphQL API
- Test backwards compatibility with existing scaling groups
- Verify options are correctly used by Manager termination logic

## Acceptance Criteria

- [ ] Health check termination options can be configured per scaling group
- [ ] GraphQL API supports CRUD operations for health check options
- [ ] Default values work correctly for existing scaling groups
- [ ] Validation prevents invalid configurations
- [ ] Documentation is complete and clear
- [ ] Tests verify all functionality

## Example Configurations

### Model Serving (Readiness-focused)

```json
{
"health_check_termination": {
"enabled": true,
"check_readiness": true,
"check_liveness": false,
"grace_period_seconds": 60,
"max_consecutive_failures": 3,
"check_interval_seconds": 30
}
}
```

### Batch Processing (Liveness-focused)

```json
{
"health_check_termination": {
"enabled": true,
"check_readiness": false,
"check_liveness": true,
"grace_period_seconds": 300,
"max_consecutive_failures": 5,
"check_interval_seconds": 60
}
}
```

### Interactive Sessions (Both checks)

```json
{
"health_check_termination": {
"enabled": true,
"check_readiness": true,
"check_liveness": true,
"grace_period_seconds": 120,
"max_consecutive_failures": 3,
"check_interval_seconds": 30
}
}
```

## Reference Code

- `src/ai/backend/manager/models/scaling_group.py` - ScalingGroupRow model
- `src/ai/backend/manager/api/gql/scaling_group.py` - GraphQL API (if exists)

## Dependencies

- This story should be implemented before or in parallel with BA-3020 (Manager termination logic)
- Manager termination logic will consume these options

JIRA Issue: BA-3021

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.