API: Add steps to TaskTemplate for spawner-driven multi-step pipelines

🤖 Axon Agent @gjkim42

Summary

TaskSpawner currently creates exactly one Task per work item. But complex work items — GitHub issues, Jira tickets, cron-triggered workflows — often need a sequence of steps (investigate → implement → test → create PR) rather than cramming everything into one enormous prompt. This proposal extends TaskTemplate with an optional steps field that creates a dependsOn chain of Tasks per work item, bringing the power of example 07 (task pipelines) into spawner-driven workflows.

Problem

1. One Task per work item is fragile for complex work

The maintainer observed this directly in #400:

If there are a lot of work to do, coding agents forget something sometimes. (E.g. Like opening a PR)
Do we have to support multi run, multi prompt feature to get more deterministic output?

Today, axon-workers.yaml packs a 77-line prompt into a single Task that must: check for existing PRs, read review comments, understand the diff, make incremental changes, self-review, ensure CI passes, and manage labels. When agents lose context mid-task, they skip steps.

2. Static pipelines don't compose with dynamic discovery

Example 07-task-pipeline shows that dependsOn + {{.Deps}} templating works beautifully — but only for manually-created Tasks with known names. A TaskSpawner can't use this pattern because:

• It doesn't know task names at config time (names are generated as <spawner>-<item.ID>)
• The DependsOn field on TaskTemplate references static names, not dynamically generated ones
• There's no way to create multiple Tasks per work item

3. Existing proposals solve different problems

Proposed API Change

Extend TaskTemplate with an optional steps field:

type TaskTemplate struct {
    // ... existing fields (type, credentials, model, image, workspaceRef, etc.) ...

    // Steps defines an ordered sequence of prompt templates that the spawner
    // expands into a dependsOn chain of Tasks for each work item.
    // When set, promptTemplate is ignored and each step's promptTemplate
    // is used instead. All steps inherit the TaskTemplate's shared fields
    // (type, credentials, model, workspaceRef, agentConfigRef, podOverrides).
    // A step may override type, model, or agentConfigRef for specialization.
    //
    // Tasks are named: <spawner>-<item.ID>-<step.name>
    // Each step (after the first) gets dependsOn: [<spawner>-<item.ID>-<prev.name>]
    //
    // +optional
    Steps []TaskStep `json:"steps,omitempty"`
}

type TaskStep struct {
    // Name identifies this step. Used in Task naming and dependency wiring.
    // Must be unique within the steps list.
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Pattern="^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
    Name string `json:"name"`

    // PromptTemplate is the Go text/template for this step's prompt.
    // Has access to all standard work item variables plus {{.Deps}}
    // for upstream step results (same as example 07).
    // +kubebuilder:validation:Required
    PromptTemplate string `json:"promptTemplate"`

    // Type optionally overrides the agent type for this step.
    // +optional
    Type string `json:"type,omitempty"`

    // Model optionally overrides the model for this step.
    // +optional
    Model string `json:"model,omitempty"`

    // AgentConfigRef optionally overrides the AgentConfig for this step.
    // +optional
    AgentConfigRef *AgentConfigReference `json:"agentConfigRef,omitempty"`
}

Validation Rules

XValidation: "!(has(self.steps) && has(self.promptTemplate))",
    message: "steps and promptTemplate are mutually exclusive"
XValidation: "has(self.steps) || has(self.promptTemplate)",
    message: "either steps or promptTemplate must be set"

How It Works

When a TaskSpawner with steps discovers work item #42, the spawner creates N Tasks with automatic dependsOn wiring instead of one:

axon-workers-42-investigate  (no deps)
    ↓
axon-workers-42-implement    (dependsOn: [axon-workers-42-investigate])
    ↓
axon-workers-42-review       (dependsOn: [axon-workers-42-implement])

Each subsequent step can reference the previous step's results via {{.Deps}}, exactly like example 07's static pipeline — because the spawner generates the same dependsOn structure.

Spawner Logic Change (in cmd/axon-spawner/main.go)

The change to runCycleWithSource is small. Currently:

// For each new item, create ONE task:
task := &axonv1alpha1.Task{Name: taskName, Spec: ...}
cl.Create(ctx, task)

With steps:

// For each new item, create N tasks with dependsOn chain:
if len(ts.Spec.TaskTemplate.Steps) > 0 {
    var prevTaskName string
    for _, step := range ts.Spec.TaskTemplate.Steps {
        stepTaskName := fmt.Sprintf("%s-%s-%s", ts.Name, item.ID, step.Name)
        task := &axonv1alpha1.Task{
            Name: stepTaskName,
            Spec: axonv1alpha1.TaskSpec{
                Type:        coalesce(step.Type, ts.Spec.TaskTemplate.Type),
                Model:       coalesce(step.Model, ts.Spec.TaskTemplate.Model),
                Prompt:      renderPrompt(step.PromptTemplate, item),
                Credentials: ts.Spec.TaskTemplate.Credentials,
                // ... other shared fields ...
            },
        }
        if prevTaskName != "" {
            task.Spec.DependsOn = []string{prevTaskName}
        }
        cl.Create(ctx, task)
        prevTaskName = stepTaskName
    }
}

The existing TaskReconciler already handles everything else: dependsOn resolution, {{.Deps}} template rendering, branch locking, output capture, and TTL cleanup.

Concrete Example: Improved axon-workers

The current 77-line monolithic prompt in self-development/axon-workers.yaml could become:

apiVersion: axon.io/v1alpha1
kind: TaskSpawner
metadata:
  name: axon-workers
spec:
  when:
    githubIssues:
      labels: [actor/axon]
      excludeLabels: [axon/needs-input]
  maxConcurrency: 3
  taskTemplate:
    type: claude-code
    model: opus
    credentials:
      type: oauth
      secretRef:
        name: axon-credentials
    workspaceRef:
      name: axon-agent
    agentConfigRef:
      name: axon-dev-agent
    branch: "axon-task-{{.Number}}"
    ttlSecondsAfterFinished: 3600
    steps:
      - name: investigate
        promptTemplate: |
          Investigate issue #{{.Number}}: {{.Title}}
          {{.Body}}
          {{if .Comments}}Comments: {{.Comments}}{{end}}

          Check if a PR already exists for branch axon-task-{{.Number}}.
          If yes, read all review comments and the existing diff.
          If no, analyze the issue and plan the fix.

          Output your findings and plan.

      - name: implement
        promptTemplate: |
          Implement the fix for issue #{{.Number}}: {{.Title}}

          Investigation results:
          {{index .Deps "investigate" "Results"}}

          Set up: git fetch --unshallow || true; git fetch origin main; git rebase origin/main

          Make the code changes, write tests, and commit.
          Push to origin axon-task-{{.Number}}.

      - name: review-and-pr
        promptTemplate: |
          Review and finalize the work for issue #{{.Number}}.

          Implementation results:
          Branch: {{index .Deps "implement" "Results" "branch"}}

          1. /review the changes. If issues found, fix them, commit, push.
          2. Create or update the PR with labels generated-by-axon and ok-to-test.
          3. Ensure CI passes.
          4. Add axon/needs-input label to the issue.
  pollInterval: 1m

Why this is better than the monolithic prompt:

1. Each step has a focused, achievable goal — agents don't lose context
2. Steps have clear success criteria — if investigation fails, implementation doesn't start
3. Different models per step — use sonnet for investigation, opus for implementation
4. Visible progress — axon get tasks shows which step each issue is on
5. Granular retry — a failed implementation step doesn't re-run investigation
6. Cost efficiency — each step gets fresh context, avoiding the exponential token growth of long conversations

Concurrency & Task Budget Considerations

• maxConcurrency should count all active Tasks across all steps, not just first-step Tasks. This is already how it works — the spawner counts all Tasks labeled axon.io/taskspawner: <name>.
• maxTotalTasks should count per work item (not per step), since steps are an implementation detail. The spawner increments newTasksCreated once per work item, even though it creates N Tasks.
• existingTasks check: a work item is considered "already created" if its first step task exists (<spawner>-<item.ID>-<first-step-name>).
• TTL: ttlSecondsAfterFinished applies to each step's Task individually. The spawner considers the pipeline complete when the last step reaches a terminal phase.

Branch Locking

All steps share the same branch value. The existing BranchLocker ensures only one step runs at a time on a given branch, which is exactly the desired behavior — steps should execute sequentially, not in parallel.

Backward Compatibility

• steps is optional. Existing TaskSpawners with promptTemplate continue to work unchanged.
• steps and promptTemplate are mutually exclusive (validated).
• Generated Tasks are standard Tasks with standard dependsOn — all existing features (metrics, output capture, TTL, CLI) work automatically.
• No changes to the Task CRD, TaskReconciler, or agent images.

Implementation Scope

Estimated: ~50 lines of new types + ~40 lines of spawner logic + tests.

Related

• #400 — Multi-run feature (this proposal is the concrete API design for that idea)
• #328 — Orchestrator pattern (this brings orchestration to spawner-driven workflows)
• #283 — taskCompletion trigger (complementary — chains spawners; this chains steps within a spawner)
• #314 — TaskSet (complementary — fans out across repos; this fans out across steps)
• Example 07 — task pipeline (this proposal reuses the same dependsOn + {{.Deps}} mechanism)
• self-development/axon-workers.yaml — real-world monolithic prompt that would benefit from steps

Start with api/v1alpha1/taskspawner_types.go and the task-creation loop in cmd/axon-spawner/main.go; then inspect internal/source/prompt.go and the existing TaskSpawner tests. Done means the API and generated CRD accept validated steps, each work item produces a correctly wired dependsOn chain, existing promptTemplate behavior remains compatible, and tests cover the new flow.