Enhanced Background Task State Management with Redis
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Overview
Enhance the background task (bgtask) state management system to support granular state tracking using Redis HSET operations instead of just start/done events.
## Current State
- Background tasks currently emit `start` event at beginning and `done` event at completion
- Intermediate states are reported via `updated` event through reporter
- Limited visibility into task progress and individual subtask states
## Proposed Enhancement
Implement comprehensive state tracking using Redis HSET for better observability and management of background tasks.
### Requirements
#### 1. Overall Task Summary Storage
Store task summary information using Redis HSET at `bgtask:{task_id}:status`:
```
HSET bgtask:{task_id}:status
pending_count 5
success_count 2
failure_count 1
keys '{"upload": "success", "process": "started", "notify": "pending"}'
created_at 1726654800.0
```
Fields:
- `pending_count`: Number of pending subtasks
- `success_count`: Number of successfully completed subtasks
- `failure_count`: Number of failed subtasks
- `keys`: JSON string mapping subtask keys to their current status
- `created_at`: Unix timestamp when the overall task was created
#### 2. Individual Subtask Detail Storage
Store detailed information for each subtask using Redis HSET at `bgtask:{task_id}:key:{key`}:
```
HSET bgtask:{task_id}:key:{key}
task_id "550e8400-e29b-41d4-a716-446655440000"
key "upload"
status "success"
created_at 1726654800.0
updated_at 1726654850.0
```
Fields:
- `task_id`: UUID of the parent task
- `key`: Subtask identifier (e.g., "upload", "process", "notify")
- `status`: Current status ("pending", "started", "success", "failure")
- `created_at`: Unix timestamp when subtask was created
- `updated_at`: Unix timestamp of last status update
#### 3. Future Considerations
- Retry mechanism fields will be addressed in a separate implementation (not in scope for this epic)
## Benefits
- Real-time visibility into task progress with aggregated counts
- Granular tracking of individual subtask states
- Better debugging capabilities with detailed timestamps
- Ability to query both summary and detailed views
- Foundation for future retry mechanism implementation
## Acceptance Criteria
- [ ] Implement two-level HSET storage structure (summary + details)
- [ ] Replace event-based state updates with HSET operations
- [ ] Maintain atomic counter updates for pending/success/failure counts
- [ ] Store subtask status mapping as JSON in summary
- [ ] Track creation and update timestamps for all entities
- [ ] Maintain backward compatibility where needed
- [ ] Add appropriate logging and monitoring
## Technical Notes
- Implementation location: `src/ai/backend/common/bgtask/`
- Existing components: bgtask manager, reporter
- Storage backend: Redis HSET
- Key structure:
- Summary: `bgtask:{task_id}:status`
- Details: `bgtask:{task_id}:key:{key`}
JIRA Issue: BA-2409
Contributor guide
Assessment
This issue has not been assessed yet.