lablup / lablup/backend.ai

Enhanced Background Task State Management with Redis

Open
#5,931 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Enhance the background task (bgtask) state management system to support granular state tracking using Redis HSET operations instead of just start/done events.

## Current State

- Background tasks currently emit `start` event at beginning and `done` event at completion
- Intermediate states are reported via `updated` event through reporter
- Limited visibility into task progress and individual subtask states

## Proposed Enhancement

Implement comprehensive state tracking using Redis HSET for better observability and management of background tasks.

### Requirements

#### 1. Overall Task Summary Storage

Store task summary information using Redis HSET at `bgtask:{task_id}:status`:

```
HSET bgtask:{task_id}:status
pending_count 5
success_count 2
failure_count 1
keys '{"upload": "success", "process": "started", "notify": "pending"}'
created_at 1726654800.0
```

Fields:

- `pending_count`: Number of pending subtasks
- `success_count`: Number of successfully completed subtasks
- `failure_count`: Number of failed subtasks
- `keys`: JSON string mapping subtask keys to their current status
- `created_at`: Unix timestamp when the overall task was created

#### 2. Individual Subtask Detail Storage

Store detailed information for each subtask using Redis HSET at `bgtask:{task_id}:key:{key`}:

```
HSET bgtask:{task_id}:key:{key}
task_id "550e8400-e29b-41d4-a716-446655440000"
key "upload"
status "success"
created_at 1726654800.0
updated_at 1726654850.0
```

Fields:

- `task_id`: UUID of the parent task
- `key`: Subtask identifier (e.g., "upload", "process", "notify")
- `status`: Current status ("pending", "started", "success", "failure")
- `created_at`: Unix timestamp when subtask was created
- `updated_at`: Unix timestamp of last status update

#### 3. Future Considerations

- Retry mechanism fields will be addressed in a separate implementation (not in scope for this epic)

## Benefits

- Real-time visibility into task progress with aggregated counts
- Granular tracking of individual subtask states
- Better debugging capabilities with detailed timestamps
- Ability to query both summary and detailed views
- Foundation for future retry mechanism implementation

## Acceptance Criteria

- [ ] Implement two-level HSET storage structure (summary + details)
- [ ] Replace event-based state updates with HSET operations
- [ ] Maintain atomic counter updates for pending/success/failure counts
- [ ] Store subtask status mapping as JSON in summary
- [ ] Track creation and update timestamps for all entities
- [ ] Maintain backward compatibility where needed
- [ ] Add appropriate logging and monitoring

## Technical Notes

- Implementation location: `src/ai/backend/common/bgtask/`
- Existing components: bgtask manager, reporter
- Storage backend: Redis HSET
- Key structure:
- Summary: `bgtask:{task_id}:status`
- Details: `bgtask:{task_id}:key:{key`}

JIRA Issue: BA-2409

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.