Implement History Tracking System for Scheduler, Deployment, and Route Operations
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Overview
Implement comprehensive history tracking system for scheduling, deployment, and routing operations in Backend.AI to improve observability and debugging capabilities.
## Goals
1. Track detailed history of scheduler operations (schedule, preparing, creating phases)
1. Implement append-only history with last state updates
1. Add history tracking for deployment operations
1. Add history tracking for route operations
1. Provide querying and monitoring capabilities
## Schema Design for Scheduler History
### Main History Record
- `session_id`: Session identifier
- `scheduling_step`: Current scheduling phase (schedule/preparing/creating)
- `status`: Overall status (progressing/success/failure)
- `error_code`: Optional error code
- `message`: Final message for the step
- `retry_count`: Number of retries for this step
- `started_at`: Step start timestamp
- `ended_at`: Step end timestamp
### Sub-phase Results
- `phase_name`: Sub-phase name
- `kernel_id`: Optional kernel ID (None for batch operations)
- `status`: Sub-phase status
- `message`: Final message
- `error_code`: Optional error code
- `started_at`: Phase start timestamp
- `ended_at`: Phase end timestamp
## Implementation Strategy
1. Start with scheduler history implementation
1. Extend to deployment and route history after scheduler is complete
1. Use append-only pattern with last state updates
1. Store in database models under `src/ai/backend/manager/models/`
1. Access through repository pattern in `src/ai/backend/manager/repositories/`
## Success Criteria
- All scheduling operations are tracked with detailed history
- History is queryable for debugging and monitoring
- System maintains performance while tracking history
- Deployment and route operations have similar tracking capability
JIRA Issue: BA-2438
Contributor guide
Assessment
This issue has not been assessed yet.