lablup / lablup/backend.ai

Manager: make the event-dispatcher plugin group usable by external plugins (forward session result events + standard dependency context)

Open
#12,526 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

### Main idea

Make the `backendai_event_dispatcher_v20` plugin group a fully usable extension point for **externally installed** (pip/package-metadata) manager plugins that react to scheduler and session lifecycle outcomes. Two gaps currently prevent this, and both can be fixed with small, plugin-agnostic changes.

#### Gap 1 — Session lifecycle result events never reach event-dispatcher plugins

`SessionEventHandler` forwards events to `EventDispatcherPluginContext.handle_event` only for `SessionStartedAnycastEvent` (`src/ai/backend/manager/event_dispatcher/handlers/session.py:103`). All other lifecycle handlers — notably `handle_batch_result` (`session.py:152`, which consumes `SessionSuccessAnycastEvent` / `SessionFailureAnycastEvent` carrying `reason` and `exit_code`) and `handle_session_terminated` — update the DB and invoke webhooks but never call the plugin context.

Since plugins never receive the `EventDispatcher` object itself, they cannot self-subscribe: the forwarding calls in the handlers are their *only* delivery path. As a result, a plugin today can observe that a session started, but not how it ended — which rules out whole categories of plugins (failure analytics, operator notifications, custom post-mortem or resubmission policies reacting to scheduler/batch outcomes).

Proposed change: forward the session result/termination events to `_event_dispatcher_plugin_ctx.handle_event(...)` in the corresponding handlers, mirroring what `handle_session_started` already does. Plugins filter by `isinstance` and ignore what they don't care about, so this is backward-compatible (the intrinsic `ErrorEventDispatcher` early-returns on non-`AgentErrorEvent`).

#### Gap 2 — No standard way for event-dispatcher plugins to obtain manager dependencies

Plugin contexts initialize at the Plugins stage (Stage 4), before repositories and processors exist, so any plugin that needs them must start disabled and be re-initialized later. The only existing mechanism is a per-plugin special case in the dependency composer:

```python
# src/ai/backend/manager/dependencies/composer.py:190
for plugin_instance in plugins.event_dispatcher_plugin_ctx.plugins.values():
if isinstance(plugin_instance, ErrorEventDispatcher):
await plugin_instance.init(context={"error_log_repository": ...})
```

This requires core to import each plugin's class — impossible for an out-of-tree plugin discovered via package metadata, and contrary to the plugin architecture (the accelerator plugin group, for comparison, needs no core-side knowledge of individual plugins).

Proposed change: replace the special case with a generic late re-initialization of **all** event-dispatcher plugins using a documented standard context dict (e.g. `etcd`, `config_provider`, `repositories`, `processors`, plus the existing `error_log_repository` key for compatibility). The natural place is inside `ProcessingComposer.compose`, after processors are created and **before** `event_dispatcher.start()` — this guarantees every plugin is fully wired before the first event can be delivered, and is strictly safer than the current Stage 6.5 timing (no events flow before `start()` anyway). Each plugin picks the keys it needs and must tolerate extra ones; the context keys become the stable contract for third-party manager plugins.

#### Expected outcome

- Third-party manager plugins installed via `pip` / `scripts/install-plugin.sh` can react to session scheduling outcomes and act through the service layer (processors), with zero manager source modifications.
- Core stops importing concrete plugin classes; `dependencies/composer.py` gets simpler.
- The intrinsic `ErrorEventDispatcher` keeps working unchanged (it already reads its dependency from the context dict by key).

### Alternative ideas

- Expose the `EventDispatcher` itself in the plugin init context and let plugins `subscribe()`/`consume()` directly. More flexible, but leaks delivery-mode decisions (anycast vs broadcast, consumer-group semantics) into plugins and makes it harder to reason about which events plugins see; the forwarding approach keeps handlers as the single choke point.
- Keep per-plugin re-init wiring in the composer but key it by entrypoint name instead of `isinstance`. Avoids the class import, but core still needs to know each plugin's dependency list, so it does not scale to external plugins.
- Out-of-process integration via session `callback_url` webhooks + REST API. Works without core changes but requires separate deployment/credentials and only covers sessions that register the callback, so it is not a substitute for a proper in-process extension point.

### Anything else?

Both changes are small and have been validated locally (unit tests for the handler forwarding and the generic re-init pass, and an externally installed test plugin is discovered via `scan_entrypoint_from_package_metadata`, receives failure events, and gets the standard context). Happy to submit the PR.

Contributor guide

Open the contributing guide

Research direction

Start with src/ai/backend/manager/event_dispatcher/handlers/session.py, especially handle_batch_result and handle_session_terminated, then inspect ProcessingComposer.compose and the existing dependency wiring in src/ai/backend/manager/dependencies/composer.py. Run the mentioned handler-forwarding and generic re-initialization tests; done means external plugins receive session result events and the standard dependency context without core importing concrete plugin classes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.