anthropics / anthropics/claude-quickstarts

Cron jobs stop executing after model quota exhaustion (FailoverError)

Ouverte
#355 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
TypeScript
Étoiles
17.6k
Forks
3k
Merge moyen
5 h 13 min
PR mergées (30 j)
3

Description

## Bug Description

Cron jobs stop executing automatically after encountering `FailoverError` due to model quota exhaustion. Once the quota is restored, cron jobs do not resume - they remain in a "stuck" state until manually triggered.

## Environment

- **OpenClaw CLI**: 2026.2.6-3
- **OpenClaw Desktop**: 2026.2.6
- **OS**: macOS Darwin 25.2.0 (arm64)
- **Node**: v25.5.0

## Symptoms

1. A cron job ("系统状态检查") last executed successfully at 2026-02-06 08:00
2. The job was configured to run every 8 hours (`0 0,8,16 * * *`)
3. Subsequent trigger times (16:00, 00:00, etc.) were **not executed**
4. 4 other cron jobs have **empty execution history** (never ran)
5. `lastStatus` shows "ok" but no new runs were recorded

## Timeline

| Time | Event |
|------|-------|
| 02-06 00:00 | Cron task failed with 429 errors |
| 02-06 08:00 | Last successful execution |
| 02-06 08:24-08:31 | Heavy 429 rate limiting |
| 02-06 16:00+ | Continued 429 errors, only manual `cron.run` recorded |
| 02-07 00:00+ | Network issues (fetch failed) |

## Key Error Logs

```
2026-02-06T00:00:53.353Z [diagnostic] lane task error: lane=cron durationMs=53233
error="FailoverError: Cloud Code Assist API error (429): You have exhausted your
capacity on this model. Your quota will reset after 80h6m35s."

2026-02-06T01:08:42.596Z [diagnostic] lane task error: lane=main durationMs=13
error="FailoverError: No available auth profile for opencode (all in cooldown or unavailable)."
```

## Task State (from cron.list)

```json
{
"state": {
"nextRunAtMs": 1770624000000, // Points to future
"lastRunAtMs": 1770336000055, // 02-06 08:00
"lastStatus": "ok",
"lastDurationMs": 97785
}
}
```

## Expected Behavior

1. Even when models are unavailable, the cron scheduler should continue triggering jobs at scheduled times
2. Failed jobs should be recorded in execution history (with error info)
3. `nextRunAtMs` should correctly update to the next trigger time
4. Optional: Add retry mechanism with backoff

## Workaround

Manually trigger the job to restore scheduling:

```bash
openclaw cron run
```

## Questions

1. Why does `lastStatus` show "ok" when subsequent jobs were not executed?
2. What is the cron scheduler behavior when models are unavailable?
3. Is there a circuit breaker mechanism that prevents future triggers?
4. Are all execution attempts recorded in run history, or only successful ones?

## Steps to Reproduce

1. Configure a cron job (e.g., hourly)
2. Exhaust all configured model quotas (or simulate 429 responses)
3. Wait for the job trigger time
4. Observe whether the job is triggered and if there is an execution record

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.