Workflow: Auto-retrigger axon-workers on CI failure for agent-created PRs

🤖 Axon Agent @gjkim42

Summary

The self-development axon-workers loop has a gap between task completion and CI feedback. When an agent creates or updates a PR, it adds axon/needs-input and exits. CI runs asynchronously afterward. If CI fails, the issue remains in axon/needs-input limbo until a human manually comments /reset-worker. This proposal adds a GitHub Actions workflow that automatically re-queues the agent when CI fails on generated-by-axon PRs, closing the feedback loop without human intervention.

Problem

Current flow:

1. Issue gets actor/axon label → axon-workers spawner creates a Task
2. Agent investigates, creates/updates PR, labels it generated-by-axon + ok-to-test
3. Agent adds axon/needs-input to the issue and exits
4. CI runs asynchronously (build, verify, test, test-integration, test-e2e)
5. If CI fails: Issue stays in axon/needs-input, PR has failing checks → dead end
6. Human must notice, comment /reset-worker, wait for agent to retry

The gap: Between step 3 (agent exits) and step 4 (CI completes), there is no feedback mechanism. The agent prompt says "Make sure the PR passes all CI tests" (self-development/axon-workers.yaml:60,68), but the agent can only check for CI status synchronously — if CI hasn't started or completed before the agent's pod terminates, the failure goes unnoticed.

Evidence this is a real problem: The /reset-worker workflow (.github/workflows/reset-axon-worker.yaml) exists specifically because this manual recovery step is needed frequently enough to warrant automation tooling. But /reset-worker still requires a human to trigger it.

Proposed Solution

Add a new GitHub Actions workflow .github/workflows/retrigger-axon-worker-on-ci-failure.yaml that:

1. Triggers on check_suite completion (or workflow_run completion)
2. Filters to PRs labeled generated-by-axon
3. When CI fails, automatically:
   • Deletes the corresponding Task/axon-workers-<issueNumber> (same as /reset-worker)
   • Removes axon/needs-input from the issue (so the spawner re-discovers it)
   • Adds a comment explaining the retry reason (CI failure details)
4. Includes a retry counter to prevent infinite loops (e.g., max 2 automatic retries)

Proposed Workflow

name: Retrigger Axon Worker on CI Failure

on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]

permissions:
  contents: read
  issues: write
  pull-requests: write
  id-token: write

jobs:
  retrigger:
    if: >
      github.event.workflow_run.conclusion == 'failure' &&
      github.event.workflow_run.event == 'pull_request'
    runs-on: ubuntu-latest
    env:
      AXON_NAMESPACE: ${{ vars.AXON_NAMESPACE || 'default' }}
      GCP_PROJECT_ID: gjkim-400213
      GKE_CLUSTER_NAME: gjkim
      GKE_CLUSTER_LOCATION: asia-northeast3
      GCP_SERVICE_ACCOUNT_EMAIL: axon-core-axon-gh-action@gjkim-400213.iam.gserviceaccount.com
      GCP_WORKLOAD_IDENTITY_PROVIDER: projects/317215297044/locations/global/workloadIdentityPools/github/providers/axon
      MAX_RETRIES: "2"
    steps:
      - name: Find the associated PR and check if it's agent-generated
        id: check
        uses: actions/github-script@v7
        with:
          script: |
            // Find PRs associated with the failed workflow run
            const headSha = context.payload.workflow_run.head_sha;
            const headBranch = context.payload.workflow_run.head_branch;

            // List PRs for this branch
            const { data: prs } = await github.rest.pulls.list({
              owner: context.repo.owner,
              repo: context.repo.repo,
              head: `${context.repo.owner}:${headBranch}`,
              state: 'open',
            });

            if (prs.length === 0) {
              core.info(`No open PRs found for branch ${headBranch}`);
              core.setOutput("should_retrigger", "false");
              return;
            }

            const pr = prs[0];
            const labels = pr.labels.map(l => l.name);

            if (!labels.includes('generated-by-axon')) {
              core.info(`PR #${pr.number} is not generated-by-axon, skipping`);
              core.setOutput("should_retrigger", "false");
              return;
            }

            // Extract the linked issue number from the PR body
            const closingRegex = /\b(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+#(\d+)/gim;
            let match;
            let issueNumber = null;
            while ((match = closingRegex.exec(pr.body || "")) !== null) {
              issueNumber = Number(match[1]);
              break;
            }

            if (!issueNumber) {
              // Try extracting from branch name: axon-task-<number>
              const branchMatch = headBranch.match(/^axon-task-(\d+)$/);
              if (branchMatch) {
                issueNumber = Number(branchMatch[1]);
              }
            }

            if (!issueNumber) {
              core.info(`Could not determine issue number for PR #${pr.number}`);
              core.setOutput("should_retrigger", "false");
              return;
            }

            // Check retry count by counting bot comments with "[auto-retry]" marker
            const { data: comments } = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: issueNumber,
            });
            const retryCount = comments.filter(c =>
              c.body && c.body.includes('[axon-auto-retry]')
            ).length;

            const maxRetries = Number(process.env.MAX_RETRIES || "2");
            if (retryCount >= maxRetries) {
              core.info(`Issue #${issueNumber} has reached max auto-retries (${retryCount}/${maxRetries}), skipping`);
              core.setOutput("should_retrigger", "false");
              return;
            }

            core.info(`Will retrigger worker for issue #${issueNumber} (PR #${pr.number}, retry ${retryCount + 1}/${maxRetries})`);
            core.setOutput("should_retrigger", "true");
            core.setOutput("issue_number", String(issueNumber));
            core.setOutput("pr_number", String(pr.number));
            core.setOutput("retry_count", String(retryCount + 1));
            core.setOutput("run_url", context.payload.workflow_run.html_url);

      - name: Authenticate to Google Cloud
        if: steps.check.outputs.should_retrigger == 'true'
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ env.GCP_WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ env.GCP_SERVICE_ACCOUNT_EMAIL }}

      - name: Configure GKE credentials
        if: steps.check.outputs.should_retrigger == 'true'
        uses: google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: ${{ env.GKE_CLUSTER_NAME }}
          location: ${{ env.GKE_CLUSTER_LOCATION }}
          project_id: ${{ env.GCP_PROJECT_ID }}

      - name: Delete the existing worker task
        if: steps.check.outputs.should_retrigger == 'true'
        run: |
          set -euo pipefail
          task_name="axon-workers-${{ steps.check.outputs.issue_number }}"
          kubectl delete task.axon.io "${task_name}" -n "${AXON_NAMESPACE}" --ignore-not-found=true
          echo "Deleted task ${task_name}"

      - name: Remove axon/needs-input label and comment
        if: steps.check.outputs.should_retrigger == 'true'
        uses: actions/github-script@v7
        env:
          ISSUE_NUMBER: ${{ steps.check.outputs.issue_number }}
          PR_NUMBER: ${{ steps.check.outputs.pr_number }}
          RETRY_COUNT: ${{ steps.check.outputs.retry_count }}
          RUN_URL: ${{ steps.check.outputs.run_url }}
          MAX_RETRIES: ${{ env.MAX_RETRIES }}
        with:
          script: |
            const issueNumber = Number(process.env.ISSUE_NUMBER);
            const prNumber = Number(process.env.PR_NUMBER);
            const retryCount = process.env.RETRY_COUNT;
            const maxRetries = process.env.MAX_RETRIES;
            const runUrl = process.env.RUN_URL;

            // Remove axon/needs-input from both issue and PR
            for (const num of new Set([issueNumber, prNumber])) {
              try {
                await github.rest.issues.removeLabel({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  issue_number: num,
                  name: 'axon/needs-input',
                });
              } catch (e) {
                if (e.status !== 404) throw e;
              }
            }

            // Comment on the issue explaining the auto-retry
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: issueNumber,
              body: [
                `[axon-auto-retry] CI failed on PR #${prNumber} — automatically re-queuing worker (retry ${retryCount}/${maxRetries}).`,
                ``,
                `Failed CI run: ${runUrl}`,
                ``,
                `The agent will investigate the CI failure and attempt to fix it.`,
              ].join('\n'),
            });

Design Decisions

Why workflow_run instead of check_suite?

The workflow_run event provides the workflow name and conclusion, allowing us to specifically target CI failures. check_suite would fire for every check (including third-party checks), requiring more filtering.

Why a retry counter instead of unlimited retries?

Without a cap, a fundamentally broken PR could trigger an infinite agent→CI→fail→retry loop, wasting API credits. The [axon-auto-retry] comment marker in the issue serves as a durable counter that survives task deletion. Default of 2 retries means the agent gets 3 total attempts (1 original + 2 retries).

Why comment on the issue?

The comment serves dual purposes: (1) the [axon-auto-retry] marker enables retry counting, and (2) it creates a visible audit trail for humans reviewing the issue history. The agent's next run will see this comment in {{.Comments}} and know that CI failed previously, providing valuable context for the fix.

Interaction with /reset-worker

The /reset-worker command continues to work as a manual override. Auto-retrigger comments don't interfere with manual resets. If a human resets after auto-retries are exhausted, the agent gets additional attempts (the counter only tracks auto-retries, not manual resets).

Benefits

1. Closes the CI feedback loop automatically — no human intervention needed for CI-fixable failures
2. Agent gets CI context — the [axon-auto-retry] comment with the failed run URL appears in {{.Comments}}, so the agent knows exactly what failed
3. Bounded retries — prevents infinite loops and runaway API costs
4. Zero API changes — pure workflow addition, uses existing Axon primitives
5. Opt-in — only applies to PRs labeled generated-by-axon

Impact on Self-Development Loop

Current loop: Issue → Agent → PR → CI fails → (human intervention) → Agent retry

Proposed loop: Issue → Agent → PR → CI fails → (auto-retrigger) → Agent retry (with CI failure context) → CI passes → Human review

This transforms the self-development workflow from semi-autonomous to near-fully-autonomous for CI-fixable issues.

References

• axon-workers prompt: self-development/axon-workers.yaml:36-77
• /reset-worker workflow: .github/workflows/reset-axon-worker.yaml
• CI workflow: .github/workflows/ci.yaml
• Task dedup logic: cmd/axon-spawner/main.go:160-185
• Related: #287 (self-development resilience — different scope: cron spawner retry, not CI feedback)
• Related: #268 (webhook triggers — complementary: webhook would be the native Axon solution, this is the GitHub Actions bridge)

Start by reading .github/workflows/reset-axon-worker.yaml and .github/workflows/ci.yaml, then inspect cmd/axon-spawner/main.go around the task deduplication logic. Implement and validate the workflow against failed CI runs for generated-by-axon PRs, including bounded retries, task deletion, label removal, and an explanatory issue comment.