API: Add TaskSet CRD for coordinated fan-out agent execution across multiple targets

🤖 Axon Agent @gjkim42

Summary

Axon's README highlights "Fan out hundreds of agents across multiple repositories" as a core differentiator (README.md:44), but the current API has no first-class primitive to express this. Today, running the same agent task across N repositories requires creating N separate TaskSpawner YAML files — each with its own Workspace, credentials, and template. This manual approach doesn't scale and contradicts the "massive parallelism" positioning.

This proposal adds a TaskSet CRD that creates a set of coordinated Tasks from a single resource definition, enabling true fan-out patterns with aggregate status tracking.

Problem

1. No fan-out primitive exists

The current CRDs support two patterns:

• Task: Run one agent against one repo
• TaskSpawner: Watch one external source and create Tasks for discovered items

Neither supports the pattern: "Run this prompt against repos A, B, C, D, ..., Z in parallel and track aggregate progress." This is the exact pattern described in the README's "Fleet-Wide Refactoring" orchestration use case (README.md:389).

2. Workaround is verbose and fragile

Issue #291 proposed a "fleet migration" example that requires one TaskSpawner per repository. For 20 repos, that's 20 Workspace resources + 20 TaskSpawner resources = 40 YAML files, all nearly identical except for the repo URL. Any change to the prompt or agent config requires updating all 20 copies.

3. No aggregate status for related Tasks

When running the same operation across multiple repos, operators need to know: "How many repos succeeded? Which ones failed? Is the fleet-wide migration complete?" Currently, this requires manually querying each Task individually.

Proposed CRD: TaskSet

// TaskSetSpec defines a set of Tasks to create from a single definition.
type TaskSetSpec struct {
    // Targets defines the set of workspaces to fan out across.
    // Each target produces one Task.
    // +kubebuilder:validation:MinItems=1
    Targets []TaskSetTarget `json:"targets"`

    // TaskTemplate defines the template for all Tasks in the set.
    // The template's workspaceRef is overridden per-target.
    // +kubebuilder:validation:Required
    TaskTemplate TaskTemplate `json:"taskTemplate"`

    // MaxConcurrency limits how many Tasks run in parallel.
    // If unset or zero, all Tasks run concurrently.
    // +optional
    // +kubebuilder:validation:Minimum=0
    MaxConcurrency *int32 `json:"maxConcurrency,omitempty"`
}

// TaskSetTarget defines a single target in the fan-out set.
type TaskSetTarget struct {
    // Name is a short identifier for this target (used in Task naming
    // and as a template variable). Must be unique within the TaskSet.
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Pattern="^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
    Name string `json:"name"`

    // WorkspaceRef references an existing Workspace resource for this target.
    // Mutually exclusive with inline.
    // +optional
    WorkspaceRef *WorkspaceReference `json:"workspaceRef,omitempty"`

    // Inline defines a workspace inline (for convenience when you
    // don't want to pre-create Workspace resources).
    // Mutually exclusive with workspaceRef.
    // +optional
    Inline *WorkspaceSpec `json:"inline,omitempty"`

    // Vars are additional key-value pairs available in the promptTemplate
    // as {{index .Vars "key"}}.
    // +optional
    Vars map[string]string `json:"vars,omitempty"`
}

// TaskSetStatus tracks aggregate progress across all Tasks.
type TaskSetStatus struct {
    // Phase is the aggregate phase of the TaskSet.
    // +optional
    Phase TaskSetPhase `json:"phase,omitempty"`

    // Total is the number of targets (= total Tasks to create).
    // +optional
    Total int `json:"total,omitempty"`

    // Pending is the number of Tasks not yet started.
    // +optional
    Pending int `json:"pending,omitempty"`

    // Running is the number of Tasks currently running.
    // +optional
    Running int `json:"running,omitempty"`

    // Succeeded is the number of Tasks that completed successfully.
    // +optional
    Succeeded int `json:"succeeded,omitempty"`

    // Failed is the number of Tasks that failed.
    // +optional
    Failed int `json:"failed,omitempty"`

    // TargetStatuses provides per-target status details.
    // +optional
    TargetStatuses []TargetStatus `json:"targetStatuses,omitempty"`

    // CompletionTime is when all Tasks reached a terminal phase.
    // +optional
    CompletionTime *metav1.Time `json:"completionTime,omitempty"`

    // Message provides a human-readable summary.
    // +optional
    Message string `json:"message,omitempty"`
}

type TargetStatus struct {
    // Name is the target name.
    Name string `json:"name"`
    // TaskName is the name of the created Task.
    TaskName string `json:"taskName"`
    // Phase is the Task's current phase.
    Phase TaskPhase `json:"phase"`
    // Outputs from the completed Task.
    // +optional
    Outputs []string `json:"outputs,omitempty"`
}

Example: Fleet-Wide Migration

This is the exact use case from the README. Migrate 5 microservices from log.Printf to structured logging — one TaskSet, one prompt, five repos:

apiVersion: axon.io/v1alpha1
kind: TaskSet
metadata:
  name: structured-logging-migration
spec:
  maxConcurrency: 3  # Run 3 agents in parallel to manage API costs
  taskTemplate:
    type: claude-code
    model: claude-sonnet-4-20250514
    credentials:
      type: oauth
      secretRef:
        name: claude-credentials
    agentConfigRef:
      name: migration-config
    ttlSecondsAfterFinished: 7200
    promptTemplate: |
      Migrate this repository from log.Printf to structured logging using slog.
      Target: {{.TargetName}}

      Rules:
      - Replace all log.Printf/log.Println with slog equivalents
      - Add structured fields for key-value context
      - Preserve existing log levels where possible
      - Run tests after migration to ensure nothing breaks
      - Create a PR with the changes

  targets:
    - name: user-service
      inline:
        repo: https://github.com/myorg/user-service
        ref: main
        secretRef:
          name: github-token
    - name: order-service
      inline:
        repo: https://github.com/myorg/order-service
        ref: main
        secretRef:
          name: github-token
    - name: payment-service
      inline:
        repo: https://github.com/myorg/payment-service
        ref: main
        secretRef:
          name: github-token
    - name: notification-service
      inline:
        repo: https://github.com/myorg/notification-service
        ref: main
        secretRef:
          name: github-token
    - name: api-gateway
      inline:
        repo: https://github.com/myorg/api-gateway
        ref: main
        secretRef:
          name: github-token

What this creates:

$ axon get taskset
NAME                           PHASE     TOTAL  PENDING  RUNNING  SUCCEEDED  FAILED  AGE
structured-logging-migration   Running   5      1        3        1          0       5m

$ axon get tasks -l axon.io/taskset=structured-logging-migration
NAME                                          TYPE         PHASE      AGE
structured-logging-migration-user-service     claude-code  Succeeded  5m
structured-logging-migration-order-service    claude-code  Running    5m
structured-logging-migration-payment-service  claude-code  Running    3m
structured-logging-migration-notif-service    claude-code  Running    3m
structured-logging-migration-api-gateway      claude-code  Pending    5m

Example: Multi-Model Comparison

Run the same task with different agents/models to compare outputs:

apiVersion: axon.io/v1alpha1
kind: TaskSet
metadata:
  name: model-comparison
spec:
  taskTemplate:
    workspaceRef:
      name: my-repo
    credentials:
      type: api-key
      secretRef:
        name: anthropic-key
    promptTemplate: |
      Refactor the authentication module to use JWT tokens.
      You are running as model variant: {{index .Vars "model_name"}}
  targets:
    - name: sonnet
      vars:
        model_name: "claude-sonnet-4"
    - name: opus
      vars:
        model_name: "claude-opus-4"

(Note: the model field would need to be overridable per-target, which could be an extension.)

Example: Security Scan Across Repos

Run a security audit against all org repos:

apiVersion: axon.io/v1alpha1
kind: TaskSet
metadata:
  name: quarterly-security-scan
spec:
  maxConcurrency: 5
  taskTemplate:
    type: claude-code
    credentials:
      type: oauth
      secretRef:
        name: claude-credentials
    agentConfigRef:
      name: security-auditor
    promptTemplate: |
      Perform a security audit of this repository ({{.TargetName}}).
      Focus on: hardcoded secrets, SQL injection, XSS, insecure dependencies.
      Create a GitHub issue with findings if any vulnerabilities are found.
  targets:
    - name: frontend
      inline:
        repo: https://github.com/myorg/frontend
        ref: main
        secretRef: { name: github-token }
    - name: backend-api
      inline:
        repo: https://github.com/myorg/backend-api
        ref: main
        secretRef: { name: github-token }
    - name: admin-portal
      inline:
        repo: https://github.com/myorg/admin-portal
        ref: main
        secretRef: { name: github-token }

Template Variables

When rendering promptTemplate in a TaskSet, these variables are available:

Variable	Type	Description
{{.TargetName}}	string	The name field of the current target
{{.TargetIndex}}	int	Zero-based index of the target in the list
{{.TotalTargets}}	int	Total number of targets in the TaskSet
{{.Vars}}	map[string]string	Per-target custom variables

Implementation Approach

Controller: TaskSetReconciler

The reconciler is straightforward — it's similar to a simplified version of the Kubernetes Job controller managing Pods:

1. Create phase: For each target, create a Task resource with the rendered prompt and the target's workspace. Label each Task with axon.io/taskset: <name> and axon.io/target: <target-name>.
2. Monitor phase: Watch owned Tasks for phase transitions. Update TaskSetStatus aggregate counts.
3. Concurrency control: Only create Tasks up to maxConcurrency. When a Task completes, create the next pending one.
4. Completion: When all Tasks reach a terminal phase, set TaskSetStatus.Phase to Completed (all succeeded) or PartiallyFailed (some failed).
5. Inline workspaces: For targets with inline workspace specs, create ephemeral Workspace resources owned by the TaskSet.

TaskSet Phases

const (
    TaskSetPhasePending   TaskSetPhase = "Pending"
    TaskSetPhaseRunning   TaskSetPhase = "Running"
    TaskSetPhaseCompleted TaskSetPhase = "Completed"       // All succeeded
    TaskSetPhaseFailed    TaskSetPhase = "PartiallyFailed"  // Some failed
)

Relationship to Existing CRDs

TaskSet (new)
  ├── creates N × Task (existing)
  │     ├── each Task references a Workspace (existing or inline-created)
  │     └── each Task uses shared credentials + AgentConfig
  └── tracks aggregate status

The TaskSet controller does NOT need to understand Jobs, Pods, or agent containers. It only creates Task resources and watches their status — the existing TaskReconciler handles everything else. This keeps the implementation thin and fully composable with existing features (TTL, output capture, metrics).

Why Not Just Use TaskSpawner?

TaskSpawner is event-driven — it reacts to external triggers (GitHub issues, cron). TaskSet is declarative — you define a fixed set of targets and execute them. The differences:

Aspect	TaskSpawner	TaskSet
Trigger	External events (GitHub, cron)	Direct apply (imperative)
Targets	Dynamic (discovered from source)	Static (listed in spec)
Lifecycle	Long-running (poll loop)	One-shot (create tasks, track to completion)
Status	Aggregate discovery stats	Per-target task status + aggregate
Use case	Continuous automation	Batch operations, migrations, audits

They are complementary. A TaskSpawner could even use taskCompletion (#283) to react when a TaskSet's tasks complete.

CLI Support

# Create a TaskSet
axon create taskset structured-logging-migration -f taskset.yaml

# Watch progress
axon get taskset --watch

# View individual target results
axon get taskset structured-logging-migration -o wide

# Delete TaskSet (cascades to Tasks)
axon delete taskset structured-logging-migration

Backward Compatibility

• Purely additive: new CRD, no changes to existing resources
• TaskSet creates standard Task resources — all existing features (TTL, output capture, metrics, AgentConfig) work automatically
• No changes to spawner, agent images, or existing controllers
• Can be adopted incrementally alongside existing Task/TaskSpawner workflows

Implementation Scope

• New files: api/v1alpha1/taskset_types.go, internal/controller/taskset_controller.go
• Modified files: CRD registration, controller manager setup, CLI commands
• Estimated scope: ~300 lines of types + ~400 lines of controller + CLI support
• No new dependencies: Uses existing controller-runtime patterns

Related Issues

Issue	Relationship
#291 (new use cases: fleet migration)	TaskSet provides the API primitive that makes fleet migration a first-class operation instead of a manual YAML exercise
#283 (task chaining)	Complementary — a taskCompletion trigger could watch for TaskSet completion to trigger downstream steps
#298 (retry policy)	Complementary — individual Tasks in a TaskSet would benefit from retry policy
#310 (notifications)	Complementary — NotificationPolicy could match on axon.io/taskset label to notify on fleet-wide completion

References

• README "Massive Parallelism" claim: README.md:44
• README "Fleet-Wide Refactoring" use case: README.md:389
• Task types: api/v1alpha1/task_types.go
• TaskSpawner types: api/v1alpha1/taskspawner_types.go
• Workspace types: api/v1alpha1/workspace_types.go
• Spawner task labeling: cmd/axon-spawner/main.go:180-182
• Kubernetes precedent: Job manages Pods; TaskSet manages Tasks (same pattern, one level up)

Start by reading the existing Task and TaskSpawner CRDs and reconcilers, then review the README.md fan-out use cases at lines 44 and 389. Implement the proposed TaskSetReconciler with owned Tasks, optional inline Workspaces, concurrency limits, template variables, and aggregate status; done means all targets reach terminal status with correct counts and completion phase.