github / github/gh-aw

ASI-08: Add circuit breaker for repeatedly failing workflows

Open
#28,776 0 comments 1 reaction 2 assignees Claimed by @lpcox View on GitHub
compiler enhancement security workflows
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 46m
Merged PRs (30d)
760

Description

## Problem

Per the [OWASP Agentic Top 10 — ASI-08 (Cascading Failures & Denial-of-Wallet)](https://github.com/microsoft/agent-governance-toolkit/blob/main/docs/OWASP-COMPLIANCE.md), agentic workflows should have circuit breakers to prevent runaway execution and cost accumulation when workflows fail repeatedly.

**Current behavior:** A workflow that fails 100 consecutive times will continue to trigger and execute on every event. There is no failure budget or automatic disabling mechanism.

### Existing Failure Safeguards

| Mechanism | File | What it does | Limitation |
|-----------|------|-------------|------------|
| `stop-after` | `stop_after.go` | Time-based cutoff (e.g., "+6h") | Time-based, not failure-based |
| Concurrency | `concurrency.go` | Limits concurrent runs | Manages parallelism, not failure budget |
| Secret validation | `compiler_activation_job_builder.go` | Validates tokens exist | Checks format, not validity |

**None of these prevent repeated execution of a failing workflow.**

## Parent Issue

Part of #28770 (OWASP Agentic Top 10 Compliance Evaluation)

## Proposed Solution

### Frontmatter Configuration

Add a `circuit-breaker` field to frontmatter:

```yaml
---
circuit-breaker:
max-consecutive-failures: 5 # Open circuit after N consecutive failures
time-window: 24h # Only count failures within this window
cooldown: 1h # Time to wait before allowing retry after circuit opens
notify: true # Post workflow annotation when circuit opens
---
```

**Defaults** (when `circuit-breaker` is not specified):
- Disabled by default (backward compatible)
- Can be enabled globally via a feature flag: `features.circuit-breaker: true` (uses sensible defaults: 5 failures, 24h window, 1h cooldown)

### Implementation Architecture

#### 1. Failure Counter (GitHub Actions Artifacts)

Use workflow run artifacts to persist the failure counter across runs:

```
Run N (success) → upload artifact: {consecutive_failures: 0, last_success: }
Run N+1 (fail) → download prev artifact → upload: {consecutive_failures: 1, last_failure: }
Run N+2 (fail) → download prev artifact → upload: {consecutive_failures: 2, last_failure: }
...
Run N+5 (fail) → consecutive_failures >= max → CIRCUIT OPEN → skip activation
Run N+6 (trigger) → check cooldown → if elapsed: allow one retry (half-open)
```

#### 2. Activation Job Integration

Add a circuit breaker check step **before** the activation condition in `compiler_activation_job_builder.go`:

```go
// In activationJobBuildContext (L17-41)
type activationJobBuildContext struct {
// ... existing fields ...
circuitBreakerConfig *CircuitBreakerConfig
}
```

New step in activation job (before validate-secret):
```yaml
- name: Check circuit breaker
id: check-circuit-breaker
uses: actions/download-artifact@v4
# Download previous run's failure counter
# Compare against threshold
# Output: is_open=true/false, consecutive_failures=N
```

Activation condition becomes:
```yaml
if: >-
${{ steps.check-circuit-breaker.outputs.is_open != 'true' &&
}}
```

#### 3. Post-Execution Counter Update

Add a step at the end of the agent job to update the failure counter:

```yaml
- name: Update circuit breaker counter
if: always()
run: |
if [ "${{ job.status }}" = "success" ]; then
echo '{"consecutive_failures": 0, "last_success": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}' > circuit-breaker-state.json
else
PREV=${{ steps.check-circuit-breaker.outputs.consecutive_failures }}
echo "{\"consecutive_failures\": $((PREV + 1)), \"last_failure\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" > circuit-breaker-state.json
fi
- uses: actions/upload-artifact@v4
if: always()
with:
name: circuit-breaker-state
path: circuit-breaker-state.json
```

#### 4. Circuit Breaker States

Follow the standard circuit breaker pattern:

```
CLOSED (normal) ──[N consecutive failures]──→ OPEN (blocking)

[cooldown elapsed]


HALF-OPEN (probe)
│ │
[success] [failure]
│ │
▼ ▼
CLOSED OPEN
```

### Key Files to Modify

| File | Change |
|------|--------|
| `pkg/workflow/frontmatter_types.go` | Add `CircuitBreaker *CircuitBreakerConfig` field |
| `pkg/workflow/circuit_breaker.go` | New file: config parsing, step generation |
| `pkg/workflow/compiler_activation_job_builder.go` | Add circuit breaker check step before activation |
| `pkg/workflow/compiler_yaml_main_job.go` | Add counter update step after agent execution |
| `pkg/parser/schemas/` | Add circuit-breaker to frontmatter JSON schema |
| `actions/setup/sh/check_circuit_breaker.sh` | Runtime script for state checking |

### Pattern to Follow

Follow the same pattern as `stop_after.go`:
- `extractCircuitBreakerConfig()` — parse frontmatter
- `generateCircuitBreakerSteps()` — generate activation/post-execution steps
- Integration in `buildActivationJob()` and main job builder

## Acceptance Criteria

- [ ] `circuit-breaker` frontmatter field parsed and validated
- [ ] Circuit breaker check runs in activation job before agent execution
- [ ] Failure counter persisted via artifacts across workflow runs
- [ ] Circuit opens after N consecutive failures within time window
- [ ] Half-open state allows one retry after cooldown period
- [ ] Workflow annotation posted when circuit opens/closes
- [ ] Feature flag `features.circuit-breaker: true` enables with defaults
- [ ] Backward compatible — no behavior change for existing workflows
- [ ] Unit tests for config parsing, state transitions, step generation
- [ ] Documentation for circuit-breaker frontmatter configuration

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.