lablup / lablup/backend.ai

Add Plugin System for Background Task Event Hooks and Webhooks

Open
#6,431 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## User Story

As a system integrator or developer, I want to register custom event handlers (webhooks, event emitters, plugins) for background task state changes, so that I can integrate background task lifecycle with external systems and trigger custom actions based on task progress.

## Context

This story builds upon BA-2409 which implements enhanced state management for background tasks using Redis HSET. With granular state tracking in place, we can now enable extensible plugin mechanisms to react to state changes.

## Functional Requirements

### 1. Plugin Interface Definition

Define a clear plugin interface that can be implemented by various handlers:

```python
class BackgroundTaskPlugin(Protocol):
async def on_task_created(self, task_id: str, context: dict) -> None:
"""Called when a new background task is created"""

async def on_task_started(self, task_id: str, context: dict) -> None:
"""Called when a task starts execution"""

async def on_subtask_updated(self, task_id: str, key: str, status: str, context: dict) -> None:
"""Called when a subtask status changes"""

async def on_task_completed(self, task_id: str, success_count: int, failure_count: int, context: dict) -> None:
"""Called when entire task completes"""

async def on_task_failed(self, task_id: str, error: Exception, context: dict) -> None:
"""Called when task fails"""
```

### 2. Plugin Registration Mechanism

Implement a plugin registry that allows:

- Dynamic plugin registration at runtime
- Configuration-based plugin loading
- Plugin enable/disable control
- Plugin execution order specification

### 3. Built-in Plugin Implementations

Provide reference implementations for common use cases:

#### Webhook Plugin

- Send HTTP POST requests to configured endpoints on task state changes
- Support configurable retry logic
- Include authentication headers (Bearer token, API key, etc.)
- Payload customization via templates

#### Event Emitter Plugin

- Publish events to event bus/message queue (e.g., Redis pub/sub, NATS, Kafka)
- Support filtering by task types or specific events
- Configurable event payload format

### 4. Plugin Execution Strategy

- Execute plugins asynchronously without blocking task execution
- Implement error isolation (plugin failure should not affect task execution)
- Support timeout configuration for plugin execution
- Log plugin execution results for debugging

### 5. Configuration Schema

Define configuration schema for plugins:

```yaml
background_task:
plugins:
- name: webhook_notifier
type: webhook
enabled: true
config:
url: https://example.com/api/task-events
auth_type: bearer
auth_token: ${WEBHOOK_TOKEN}
events: [task_completed, task_failed]
retry_count: 3

- name: event_publisher
type: event_emitter
enabled: true
config:
channel: bgtask.events
events: [subtask_updated, task_completed]
```

## Technical Requirements

1. **Non-blocking Execution**: Plugins must not block task execution
1. **Error Isolation**: Plugin failures must not propagate to task execution
1. **Observability**: Log all plugin invocations and their outcomes
1. **Testing**: Provide mock plugins for testing purposes
1. **Documentation**: Document plugin interface and provide implementation examples

## Acceptance Criteria

- [ ] Plugin interface (Protocol) is defined with all lifecycle hooks
- [ ] Plugin registry supports registration, enable/disable, and execution order
- [ ] Webhook plugin implementation with retry and authentication support
- [ ] Event emitter plugin implementation for message queue integration
- [ ] Configuration schema supports plugin definition in TOML/YAML
- [ ] Plugin execution is asynchronous and non-blocking
- [ ] Plugin errors are isolated and logged appropriately
- [ ] Unit tests cover plugin interface, registry, and built-in implementations
- [ ] Documentation includes plugin development guide and examples

## Implementation Notes

- Implementation location: `src/ai/backend/common/bgtask/plugins/`
- Core integration point: Existing bgtask manager and reporter
- Plugin discovery: Configuration-based + programmatic registration
- Default plugins should be opt-in via configuration

## Dependencies

- Depends on BA-2409 (Enhanced Background Task State Management) for granular state tracking

## Related Issues

- Epic: BA-2409 - Enhanced Background Task State Management with Redis

JIRA Issue: BA-2849

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.