github / github/gh-aw

AI-credits rate-limit detector false-positives on echoed MCP tool results, and self-seeds via its own issue title

Closed
#61,141 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

### What happened

A weekly scheduled workflow (`engine: claude`, WIF auth to `api.anthropic.com`) filed `[aw] hit AI credits rate limit` three weeks running. Every one of those runs concluded `success`, every job was green, and each published its `create-discussion` safe output.

There was no 429. The api-proxy ledger in the `agent` artifact (`sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl`) records every model request with its HTTP status: **66 requests across the three runs, all `200`**. No `429`, `HTTP 429`, `status=429` or `Too Many Requests` string exists anywhere in any of the three runs' artifacts.

The conclusion job shows the signal has exactly one source:

```
GH_AW_AI_CREDITS_RATE_LIMIT_ERROR: true
[ai-credits] rateLimitSignal source=env(GH_AW_AI_CREDITS_RATE_LIMIT_ERROR)
AI credits rate-limit error: true
Max AI credits exceeded (harness budget abort): false
```

### Root cause

`actions/setup/js/parse_mcp_gateway_log.cjs:47-51`:

```js
const AI_CREDITS_RATE_LIMIT_PATTERNS = [
/ai[\s_-]*credits?.*(?:rate[\s-]*limit|limit exceeded|budget exceeded|exceeded)/i,
/(?:rate[\s-]*limit|too many requests).*(?:ai[\s_-]*credits?)/i,
/\b429\b.*(?:rate[\s-]*limit|too many requests|ai[\s_-]*credits?)/i,
];
```

These run over `gateway.log` / `stderr.log` / `gateway.md`. Those logs echo **MCP tool results verbatim**, and JSON-escape newlines as literal `\n` — so an entire multi-paragraph commit message, issue body or discussion body arrives as a single physical line. Combined with an unanchored pattern that has no bound on the `.*` gap, any single log line carrying an AI-credits token anywhere near a rate-limit token fires the detector.

Replaying pattern 0 against the three real gateway logs:

| Run | Gateway log line | MCP tool | Matched span | What matched |
|---|---|---|---|---|
| 1 | L2192 | `list_commits` | **19,592 chars** | a commit message mentioning `ai_credits_total`, then ~19 KB later the words `a rate limit` |
| 2 | L1826 | `list_issues` | 21 chars | the issue title `[aw] hit AI credits rate limit` |
| 3 | L1729 | `list_issues` | 21 chars | same title |

### Two things make this worse than a cosmetic false positive

**1. It is self-seeding.** `actions/setup/js/handle_agent_failure.cjs:335` generates the title:

```js
if (options.aiCreditsRateLimitError) return `[aw] ${workflowName} hit AI credits rate limit`;
```

Any workflow that reads its own repo's issues — including `githubnext/agentics`' `weekly-research`, whose prompt says "Read selections of the latest code, issues and PRs" — reads that title back through `list_issues` on the next run and re-fires the detector. One false positive guarantees the next one. Runs 2 and 3 above are exactly that loop.

**2. It overrides a successful conclusion.** `handle_agent_failure.cjs:3669` lists `!aiCreditsRateLimitError` in the early-return gate, so a true flag forces failure handling even when `agentConclusion === "success"` and valid safe outputs were produced. Compare `:3621`, where the genuine engine-429 detector is correctly gated on `agentConclusion === "failure"`.

Users then act on the failure issue's text — `The Copilot API returned a rate limit response (HTTP 429)` (`actions/setup/md/ai_credits_rate_limit_throttle.md`) — which is emitted for every engine, so on a non-Copilot engine it misdirects the diagnosis on top of being wrong about there being a throttle at all.

There is no per-category suppression; `report-failure-as-issue: false` is the only knob, and it disables genuine failure reporting too.

### Reproduces on `main`

`AI_CREDITS_RATE_LIMIT_PATTERNS` is byte-identical on `main` and `v0.88.7`, so this is not fixed by upgrading. Minimal repro without running a workflow:

```js
const p = /ai[\s_-]*credits?.*(?:rate[\s-]*limit|limit exceeded|budget exceeded|exceeded)/i;
p.test('... tool_result ... "title":"[aw] Weekly Research hit AI credits rate limit" ...'); // true
```

### Suggested directions

- Bound the gap between the two tokens (e.g. `.{0,80}`), so a 19 KB span cannot match.
- Better: derive the signal from proxy/gateway **status** fields rather than from log text that echoes tool-result payloads. The api-proxy ledger already records per-request `status` per provider — cross-checking it would have ruled all three of these out immediately.
- At minimum, exclude lines that are MCP tool-result echoes from the scan, and stop treating `aiCreditsRateLimitError` as sufficient to override an otherwise successful conclusion.
- Independently: make the `ai_credits_rate_limit_throttle.md` wording engine-aware instead of naming the Copilot API unconditionally.

Contributor guide

Open the contributing guide

Research direction

Start with the minimal regex reproduction, then inspect actions/setup/js/parse_mcp_gateway_log.cjs, actions/setup/js/handle_agent_failure.cjs, and actions/setup/md/ai_credits_rate_limit_throttle.md at the cited locations. Compare gateway log scanning with the api-proxy token-usage.jsonl status fields and the genuine engine-429 gate. Done means echoed issue or commit text no longer triggers the signal, successful runs are not overridden, and the failure wording is engine-aware.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, javascript
Domain
ci-cd, devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.