lablup / lablup/backend.ai

Implement History Tracking System for Scheduler, Deployment, and Route Operations

Open
#5,966 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Implement comprehensive history tracking system for scheduling, deployment, and routing operations in Backend.AI to improve observability and debugging capabilities.

## Goals

1. Track detailed history of scheduler operations (schedule, preparing, creating phases)
1. Implement append-only history with last state updates
1. Add history tracking for deployment operations
1. Add history tracking for route operations
1. Provide querying and monitoring capabilities

## Schema Design for Scheduler History

### Main History Record

- `session_id`: Session identifier
- `scheduling_step`: Current scheduling phase (schedule/preparing/creating)
- `status`: Overall status (progressing/success/failure)
- `error_code`: Optional error code
- `message`: Final message for the step
- `retry_count`: Number of retries for this step
- `started_at`: Step start timestamp
- `ended_at`: Step end timestamp

### Sub-phase Results

- `phase_name`: Sub-phase name
- `kernel_id`: Optional kernel ID (None for batch operations)
- `status`: Sub-phase status
- `message`: Final message
- `error_code`: Optional error code
- `started_at`: Phase start timestamp
- `ended_at`: Phase end timestamp

## Implementation Strategy

1. Start with scheduler history implementation
1. Extend to deployment and route history after scheduler is complete
1. Use append-only pattern with last state updates
1. Store in database models under `src/ai/backend/manager/models/`
1. Access through repository pattern in `src/ai/backend/manager/repositories/`

## Success Criteria

- All scheduling operations are tracked with detailed history
- History is queryable for debugging and monitoring
- System maintains performance while tracking history
- Deployment and route operations have similar tracking capability

JIRA Issue: BA-2438

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.