Add Plugin System for Background Task Event Hooks and Webhooks
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## User Story
As a system integrator or developer, I want to register custom event handlers (webhooks, event emitters, plugins) for background task state changes, so that I can integrate background task lifecycle with external systems and trigger custom actions based on task progress.
## Context
This story builds upon BA-2409 which implements enhanced state management for background tasks using Redis HSET. With granular state tracking in place, we can now enable extensible plugin mechanisms to react to state changes.
## Functional Requirements
### 1. Plugin Interface Definition
Define a clear plugin interface that can be implemented by various handlers:
```python
class BackgroundTaskPlugin(Protocol):
async def on_task_created(self, task_id: str, context: dict) -> None:
"""Called when a new background task is created"""
async def on_task_started(self, task_id: str, context: dict) -> None:
"""Called when a task starts execution"""
async def on_subtask_updated(self, task_id: str, key: str, status: str, context: dict) -> None:
"""Called when a subtask status changes"""
async def on_task_completed(self, task_id: str, success_count: int, failure_count: int, context: dict) -> None:
"""Called when entire task completes"""
async def on_task_failed(self, task_id: str, error: Exception, context: dict) -> None:
"""Called when task fails"""
```
### 2. Plugin Registration Mechanism
Implement a plugin registry that allows:
- Dynamic plugin registration at runtime
- Configuration-based plugin loading
- Plugin enable/disable control
- Plugin execution order specification
### 3. Built-in Plugin Implementations
Provide reference implementations for common use cases:
#### Webhook Plugin
- Send HTTP POST requests to configured endpoints on task state changes
- Support configurable retry logic
- Include authentication headers (Bearer token, API key, etc.)
- Payload customization via templates
#### Event Emitter Plugin
- Publish events to event bus/message queue (e.g., Redis pub/sub, NATS, Kafka)
- Support filtering by task types or specific events
- Configurable event payload format
### 4. Plugin Execution Strategy
- Execute plugins asynchronously without blocking task execution
- Implement error isolation (plugin failure should not affect task execution)
- Support timeout configuration for plugin execution
- Log plugin execution results for debugging
### 5. Configuration Schema
Define configuration schema for plugins:
```yaml
background_task:
plugins:
- name: webhook_notifier
type: webhook
enabled: true
config:
url: https://example.com/api/task-events
auth_type: bearer
auth_token: ${WEBHOOK_TOKEN}
events: [task_completed, task_failed]
retry_count: 3
- name: event_publisher
type: event_emitter
enabled: true
config:
channel: bgtask.events
events: [subtask_updated, task_completed]
```
## Technical Requirements
1. **Non-blocking Execution**: Plugins must not block task execution
1. **Error Isolation**: Plugin failures must not propagate to task execution
1. **Observability**: Log all plugin invocations and their outcomes
1. **Testing**: Provide mock plugins for testing purposes
1. **Documentation**: Document plugin interface and provide implementation examples
## Acceptance Criteria
- [ ] Plugin interface (Protocol) is defined with all lifecycle hooks
- [ ] Plugin registry supports registration, enable/disable, and execution order
- [ ] Webhook plugin implementation with retry and authentication support
- [ ] Event emitter plugin implementation for message queue integration
- [ ] Configuration schema supports plugin definition in TOML/YAML
- [ ] Plugin execution is asynchronous and non-blocking
- [ ] Plugin errors are isolated and logged appropriately
- [ ] Unit tests cover plugin interface, registry, and built-in implementations
- [ ] Documentation includes plugin development guide and examples
## Implementation Notes
- Implementation location: `src/ai/backend/common/bgtask/plugins/`
- Core integration point: Existing bgtask manager and reporter
- Plugin discovery: Configuration-based + programmatic registration
- Default plugins should be opt-in via configuration
## Dependencies
- Depends on BA-2409 (Enhanced Background Task State Management) for granular state tracking
## Related Issues
- Epic: BA-2409 - Enhanced Background Task State Management with Redis
JIRA Issue: BA-2849
Contributor guide
Assessment
This issue has not been assessed yet.