API: Add cost/token Prometheus metrics and BudgetPolicy for production cost governance

🤖 Axon Agent @gjkim42

Summary

Axon already captures per-task cost and token usage data (cost-usd, input-tokens, output-tokens) via the axon-capture sidecar, stored in TaskStatus.Results. However, this data is never exposed as Prometheus metrics and there is no mechanism to enforce spending limits. This makes Axon a cost black box in production — operators can see how many tasks ran, but not how much they cost.

This proposal adds (1) cost/token Prometheus metrics to the existing metrics infrastructure, and (2) a BudgetPolicy CRD for namespace or spawner-level spend limits.

Problem

1. Cost data is captured but not observable

The capture sidecar (internal/capture/usage.go) already parses agent output files for all four agent types:

// internal/capture/capture.go:86-91
agentType := os.Getenv("AXON_AGENT_TYPE")
usage := ParseUsage(agentType, usageFile)
for _, key := range []string{"cost-usd", "input-tokens", "output-tokens"} {
    if v, ok := usage[key]; ok {
        outputs = append(outputs, key+": "+v)
    }
}

These values flow into TaskStatus.Results via ResultsFromOutputs() in internal/controller/output_parser.go:44. But the existing Prometheus metrics (internal/controller/metrics.go) only expose:

• axon_task_created_total — task count by namespace/type
• axon_task_completed_total — terminal phase count
• axon_task_duration_seconds — execution time histogram
• axon_reconcile_errors_total — controller errors

No cost or token metrics exist. Operators cannot answer: "How much did we spend on Axon tasks today?" or "Which TaskSpawner is consuming the most tokens?" without manually querying every Task's .status.results map.

2. No spending limits or budget governance

The existing governance primitives are volume-based:

• maxConcurrency: limits concurrent tasks (but a single expensive task can cost more than 100 cheap ones)
• maxTotalTasks: lifetime task count limit (same problem — cost varies wildly per task)
• activeDeadlineSeconds: time-based timeout (a 5-minute task with opus can cost 10x a 30-minute task with haiku)

None of these control actual spend. An autonomous spawner processing issues with model: opus could burn through hundreds of dollars before anyone notices. This is the #1 concern enterprises raise about autonomous agents.

3. Real-world impact

In the self-development configs:

• axon-workers.yaml uses model: opus with maxConcurrency: 3 — three concurrent opus tasks processing GitHub issues. Each could cost $5-50+ depending on complexity, with no spend cap.
• axon-fake-strategist.yaml runs every 12 hours with opus — roughly $10-30/day with no budget limit.
• There's no way to detect or alert on cost anomalies (e.g., a runaway agent in an infinite loop consuming tokens).

Proposed Changes

Part 1: Cost/Token Prometheus Metrics (small, concrete)

Add metrics to internal/controller/metrics.go that emit cost and token data when tasks complete:

var (
    // taskCostUSD records the cost in USD of completed Tasks.
    taskCostUSD = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "axon_task_cost_usd_total",
            Help: "Total cost in USD of completed Tasks",
        },
        []string{"namespace", "type", "spawner", "model"},
    )

    // taskInputTokens records the total input tokens consumed by completed Tasks.
    taskInputTokens = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "axon_task_input_tokens_total",
            Help: "Total input tokens consumed by completed Tasks",
        },
        []string{"namespace", "type", "spawner", "model"},
    )

    // taskOutputTokens records the total output tokens consumed by completed Tasks.
    taskOutputTokens = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "axon_task_output_tokens_total",
            Help: "Total output tokens consumed by completed Tasks",
        },
        []string{"namespace", "type", "spawner", "model"},
    )
)

Instrumentation point — in task_controller.go, when TaskStatus.Results is populated (around line 482):

if results != nil {
    spawner := task.Labels["axon.io/taskspawner"]
    model := task.Spec.Model
    if costStr, ok := results["cost-usd"]; ok {
        if cost, err := strconv.ParseFloat(costStr, 64); err == nil {
            taskCostUSD.WithLabelValues(task.Namespace, task.Spec.Type, spawner, model).Add(cost)
        }
    }
    if inputStr, ok := results["input-tokens"]; ok {
        if tokens, err := strconv.ParseFloat(inputStr, 64); err == nil {
            taskInputTokens.WithLabelValues(task.Namespace, task.Spec.Type, spawner, model).Add(tokens)
        }
    }
    if outputStr, ok := results["output-tokens"]; ok {
        if tokens, err := strconv.ParseFloat(outputStr, 64); err == nil {
            taskOutputTokens.WithLabelValues(task.Namespace, task.Spec.Type, spawner, model).Add(tokens)
        }
    }
}

Scope: ~40 lines in metrics.go + ~20 lines in task_controller.go. No new dependencies.

Enabled use cases:

# Total spend in the last 24 hours
sum(increase(axon_task_cost_usd_total[24h]))

# Cost per spawner (which spawner is most expensive?)
sum by (spawner) (increase(axon_task_cost_usd_total[24h]))

# Average cost per task by model
sum by (model) (increase(axon_task_cost_usd_total[1h]))
/ sum by (model) (increase(axon_task_completed_total[1h]))

# Alert: spending more than $50/day
sum(increase(axon_task_cost_usd_total[24h])) > 50

# Token consumption rate (useful for API rate planning)
rate(axon_task_input_tokens_total[1h])

Part 2: BudgetPolicy CRD (medium, high-impact)

A new CRD that enables cost-based governance:

// BudgetPolicy defines spending limits for Tasks within a scope.
type BudgetPolicy struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec              BudgetPolicySpec   `json:"spec,omitempty"`
    Status            BudgetPolicyStatus `json:"status,omitempty"`
}

type BudgetPolicySpec struct {
    // Scope defines what this budget applies to.
    // +kubebuilder:validation:Required
    Scope BudgetScope `json:"scope"`

    // Limits defines the spending limits.
    // +kubebuilder:validation:Required
    Limits BudgetLimits `json:"limits"`

    // Window is the rolling time window for budget calculation.
    // Supported formats: "1h", "24h", "7d", "30d".
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Pattern="^[0-9]+(h|d)$"
    Window string `json:"window"`

    // Action defines what happens when the budget is exceeded.
    // +kubebuilder:validation:Enum=Suspend;Warn
    // +kubebuilder:default=Warn
    // +optional
    Action string `json:"action,omitempty"`
}

type BudgetScope struct {
    // Namespace scopes the budget to a specific namespace.
    // If empty, applies to the namespace where BudgetPolicy is created.
    // +optional
    Namespace string `json:"namespace,omitempty"`

    // TaskSpawnerRef scopes the budget to a specific TaskSpawner.
    // +optional
    TaskSpawnerRef *string `json:"taskSpawnerRef,omitempty"`
}

type BudgetLimits struct {
    // MaxCostUSD is the maximum total cost in USD within the window.
    // +optional
    // +kubebuilder:validation:Pattern="^[0-9]+(\\.[0-9]+)?$"
    MaxCostUSD *string `json:"maxCostUSD,omitempty"`

    // MaxInputTokens is the maximum total input tokens within the window.
    // +optional
    MaxInputTokens *int64 `json:"maxInputTokens,omitempty"`

    // MaxOutputTokens is the maximum total output tokens within the window.
    // +optional
    MaxOutputTokens *int64 `json:"maxOutputTokens,omitempty"`
}

type BudgetPolicyStatus struct {
    // CurrentSpend tracks actual spending in the current window.
    // +optional
    CurrentCostUSD string `json:"currentCostUSD,omitempty"`

    // CurrentInputTokens tracks actual input tokens in the current window.
    // +optional
    CurrentInputTokens int64 `json:"currentInputTokens,omitempty"`

    // CurrentOutputTokens tracks actual output tokens in the current window.
    // +optional
    CurrentOutputTokens int64 `json:"currentOutputTokens,omitempty"`

    // Exceeded is true when any limit has been reached.
    // +optional
    Exceeded bool `json:"exceeded,omitempty"`

    // LastUpdated is when the budget status was last calculated.
    // +optional
    LastUpdated *metav1.Time `json:"lastUpdated,omitempty"`
}

Example usage:

# Limit total daily spend across all tasks in the namespace
apiVersion: axon.io/v1alpha1
kind: BudgetPolicy
metadata:
  name: daily-budget
spec:
  scope: {}  # namespace-scoped (uses the namespace where this resource lives)
  limits:
    maxCostUSD: "100"
  window: "24h"
  action: Suspend  # auto-suspend TaskSpawners when budget exceeded
---
# Limit a specific spawner's weekly spend
apiVersion: axon.io/v1alpha1
kind: BudgetPolicy
metadata:
  name: workers-weekly-budget
spec:
  scope:
    taskSpawnerRef: axon-workers
  limits:
    maxCostUSD: "500"
    maxOutputTokens: 10000000  # 10M output tokens
  window: "7d"
  action: Warn  # emit a Kubernetes Event warning but don't stop

Implementation approach for BudgetPolicy

The BudgetPolicyReconciler would:

1. Periodically query completed Tasks within the scope and window
2. Sum cost-usd, input-tokens, output-tokens from TaskStatus.Results
3. Update BudgetPolicyStatus with current totals
4. When action: Suspend and limits exceeded:
   • Set spec.suspend: true on matching TaskSpawners
   • Record a Kubernetes Event on both the BudgetPolicy and the TaskSpawner
5. When the window rolls over and spend drops below limits:
   • Clear the Exceeded flag
   • Optionally auto-resume suspended TaskSpawners (configurable)

This leverages the existing suspend field on TaskSpawner (added in #326) for enforcement.

Implementation Priority

1. Phase 1: Cost Prometheus metrics — ~60 lines of code, immediate value, no new CRDs. Enables Grafana dashboards and Prometheus alerts for cost monitoring.
2. Phase 2: BudgetPolicy CRD — New CRD + controller. Enables automated cost governance. Depends on Phase 1 data being available.

Phase 1 alone provides substantial value and can be shipped independently.

Why this matters for adoption

Every enterprise evaluation of autonomous agents asks the same question: "How do we prevent runaway costs?" The current answer with Axon is "set maxConcurrency and activeDeadlineSeconds" — but these are indirect, imprecise proxies for cost. A single 3-minute opus task can cost more than ten 30-minute sonnet tasks.

Cost metrics + budget governance makes Axon production-ready for organizations where multiple teams run autonomous agents and need chargeback, accountability, and safety guardrails.

Backward Compatibility

• Phase 1: Purely additive metrics — no API changes, no behavior changes
• Phase 2: New CRD — no changes to existing resources. BudgetPolicy is opt-in; clusters without BudgetPolicy resources behave identically to today

References

• Capture sidecar cost extraction: internal/capture/capture.go:86-91
• Usage parsing (all 4 agents): internal/capture/usage.go
• Existing Prometheus metrics: internal/controller/metrics.go
• Results stored in status: internal/controller/task_controller.go:482
• Results parsed from outputs: internal/controller/output_parser.go:44 (ResultsFromOutputs)
• TaskSpawner suspend field: api/v1alpha1/taskspawner_types.go (implemented via #326)
• Self-dev workers (opus, concurrent): self-development/axon-workers.yaml:17
• Self-dev strategist (opus, cron): self-development/axon-fake-strategist.yaml:14

Start with internal/controller/metrics.go and internal/controller/task_controller.go around line 482, then review internal/controller/output_parser.go and the TaskSpawner API. Separate the concrete metrics phase from the larger BudgetPolicy CRD and controller design. Done means cost and token usage are exposed from task results, with budget limits and enforcement behavior specified and covered for the proposed scopes.