API: Add onCompletion notification hooks to TaskSpawner for outbound event delivery on task terminal phases

🤖 Kelos Strategist Agent @gjkim42

Summary

Kelos currently has no mechanism for notifying external systems when tasks reach terminal phases (Succeeded or Failed). The only outbound reporting is GitHubReporting (taskspawner_types.go:48-53), which is limited to posting status comments back to the originating GitHub issue or PR. When a task fails silently (OOM, timeout, agent crash), the only signals are:

1. Task.Status.Phase flipping to Failed (requires polling the API)
2. Kubernetes events (ephemeral, default 1-hour retention, no external routing)
3. Prometheus metrics kelos_task_completed_total (requires monitoring stack)

None of these provide push-based notification to external systems. This is a critical gap for production deployments where task failures need immediate human attention.

Problem

Silent failures are invisible. When an agent pod OOMs or hits activeDeadlineSeconds, the agent never gets a chance to post its own status. The controller transitions the task to Failed (task_controller.go:507-512), records a metric, and moves on. No alert fires unless users have custom Prometheus alerting rules watching Kelos-specific metrics.

No way to close the loop on non-GitHub workflows. Jira-sourced and cron-sourced tasks have no reporting mechanism at all — GitHubReporting only applies to GitHub sources. When a cron task fails at 3am, nobody knows until they manually check.

Integration requires polling. External systems (Slack, PagerDuty, custom dashboards) that need to react to task completion must poll the Kubernetes API for Task status changes. This is inefficient and adds latency.

Proposal

Add an onCompletion field to TaskSpawnerSpec (and optionally TaskSpec) that configures outbound notification hooks when tasks reach terminal phases. This extends the existing GitHubReporting pattern to arbitrary destinations.

API Design

// NotificationHook defines an outbound notification destination.
type NotificationHook struct {
    // Name identifies this hook for logging and status reporting.
    // +kubebuilder:validation:Required
    Name string `json:"name"`

    // Phases specifies which terminal phases trigger this hook.
    // Defaults to both Succeeded and Failed.
    // +kubebuilder:validation:Items:Enum=Succeeded;Failed
    // +kubebuilder:default={"Succeeded","Failed"}
    // +optional
    Phases []TaskPhase `json:"phases,omitempty"`

    // Webhook sends an HTTP POST with task details to the given URL.
    // +optional
    Webhook *WebhookNotification `json:"webhook,omitempty"`
}

// WebhookNotification configures an HTTP webhook notification.
type WebhookNotification struct {
    // URL is the webhook endpoint.
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Pattern="^https?://.+"
    URL string `json:"url"`

    // SecretRef optionally references a Secret containing headers
    // (e.g., Authorization tokens) to include in the request.
    // +optional
    SecretRef *SecretReference `json:"secretRef,omitempty"`
}

// OnCompletion configures outbound notifications for task lifecycle events.
type OnCompletion struct {
    // Hooks is a list of notification destinations.
    // +optional
    Hooks []NotificationHook `json:"hooks,omitempty"`
}

Added to TaskSpawnerSpec:

type TaskSpawnerSpec struct {
    // ... existing fields ...

    // OnCompletion configures outbound notifications when spawned tasks
    // reach terminal phases.
    // +optional
    OnCompletion *OnCompletion `json:"onCompletion,omitempty"`
}

Webhook Payload

The webhook POST body would include the task's key metadata:

{
  "task": "my-task-issue-42",
  "namespace": "default",
  "spawner": "my-spawner",
  "phase": "Failed",
  "message": "Task failed",
  "agentType": "claude-code",
  "model": "claude-sonnet-4-20250514",
  "startTime": "2026-03-20T10:00:00Z",
  "completionTime": "2026-03-20T10:05:30Z",
  "outputs": ["https://github.com/org/repo/pull/123"],
  "results": {
    "cost-usd": "0.42",
    "input-tokens": "15000",
    "output-tokens": "3200"
  }
}

Implementation Location

The notification dispatch would be added to the task controller's terminal phase transition block in task_controller.go:498-513, after the phase is set and outputs are captured but before the status update is persisted. The controller would look up the originating TaskSpawner (via the kelos.dev/taskspawner label already set on spawned tasks) and execute any configured hooks.

Example Configuration

apiVersion: kelos.dev/v1alpha1
kind: TaskSpawner
metadata:
  name: issue-worker
spec:
  when:
    githubIssues:
      labels: ["kelos"]
  taskTemplate:
    type: claude-code
    credentials:
      type: api-key
      secretRef:
        name: anthropic-credentials
    workspaceRef:
      name: my-workspace
  onCompletion:
    hooks:
      - name: slack-alert-on-failure
        phases: ["Failed"]
        webhook:
          url: "https://hooks.slack.com/services/T.../B.../xxx"
      - name: audit-log
        phases: ["Succeeded", "Failed"]
        webhook:
          url: "https://internal.example.com/kelos/task-completed"
          secretRef:
            name: audit-webhook-token

Relationship to Existing Issues

• Different from #595 (Slack as source): #595 proposes Slack as an inbound trigger for spawning tasks. This proposal is about outbound notifications when tasks complete.
• Different from #687 (webhook source): #687 proposes webhooks as an inbound trigger. This is the reverse direction.
• Different from #600 (TaskStatus conditions): #600 improves the content of TaskStatus. This proposal is about delivering status to external systems.
• Different from #658 (metrics labels): #658 adds labels to Prometheus metrics. This proposal is about push-based notifications to arbitrary endpoints.
• Extends GitHubReporting pattern: The existing GitHubReporting field (taskspawner_types.go:48-53) is a narrow form of this — hardcoded to post comments on GitHub. This proposal generalizes the pattern to any HTTP endpoint.

Incremental Adoption Path

1. Phase 1: Webhook-only onCompletion hooks (as described above)
2. Phase 2: Add built-in Slack notification type with rich message formatting
3. Phase 3: Consolidate GitHubReporting into the onCompletion framework as a github hook type, deprecating the separate field

Why This Matters

• Production readiness: Teams running Kelos in production need alerting on failures without building custom Prometheus rules
• Cron/Jira parity: GitHub-sourced tasks have GitHubReporting; cron and Jira tasks have no reporting mechanism at all
• Agent crash resilience: When agents crash (OOM, timeout), they cannot self-report — only the controller can notify
• Integration without new CRDs: Enables Slack, PagerDuty, Teams, and custom system integration through a single, generic mechanism

/kind feature

Start by reading the existing GitHubReporting definition in taskspawner_types.go:48-53 and the terminal-phase transition block in task_controller.go:498-513. Determine how the proposed API, hook lookup, delivery behavior, and failure handling fit the existing controller flow; done requires an agreed design and implementation covering the stated webhook configuration and terminal phases.