anthropics / anthropics/claude-quickstarts
Cron jobs stop executing after model quota exhaustion (FailoverError)
- Langage dominant
- TypeScript
- Étoiles
- 17.6k
- Forks
- 3k
- Merge moyen
- 5 h 13 min
- PR mergées (30 j)
- 3
Description
## Bug Description
Cron jobs stop executing automatically after encountering `FailoverError` due to model quota exhaustion. Once the quota is restored, cron jobs do not resume - they remain in a "stuck" state until manually triggered.
## Environment
- **OpenClaw CLI**: 2026.2.6-3
- **OpenClaw Desktop**: 2026.2.6
- **OS**: macOS Darwin 25.2.0 (arm64)
- **Node**: v25.5.0
## Symptoms
1. A cron job ("系统状态检查") last executed successfully at 2026-02-06 08:00
2. The job was configured to run every 8 hours (`0 0,8,16 * * *`)
3. Subsequent trigger times (16:00, 00:00, etc.) were **not executed**
4. 4 other cron jobs have **empty execution history** (never ran)
5. `lastStatus` shows "ok" but no new runs were recorded
## Timeline
| Time | Event |
|------|-------|
| 02-06 00:00 | Cron task failed with 429 errors |
| 02-06 08:00 | Last successful execution |
| 02-06 08:24-08:31 | Heavy 429 rate limiting |
| 02-06 16:00+ | Continued 429 errors, only manual `cron.run` recorded |
| 02-07 00:00+ | Network issues (fetch failed) |
## Key Error Logs
```
2026-02-06T00:00:53.353Z [diagnostic] lane task error: lane=cron durationMs=53233
error="FailoverError: Cloud Code Assist API error (429): You have exhausted your
capacity on this model. Your quota will reset after 80h6m35s."
2026-02-06T01:08:42.596Z [diagnostic] lane task error: lane=main durationMs=13
error="FailoverError: No available auth profile for opencode (all in cooldown or unavailable)."
```
## Task State (from cron.list)
```json
{
"state": {
"nextRunAtMs": 1770624000000, // Points to future
"lastRunAtMs": 1770336000055, // 02-06 08:00
"lastStatus": "ok",
"lastDurationMs": 97785
}
}
```
## Expected Behavior
1. Even when models are unavailable, the cron scheduler should continue triggering jobs at scheduled times
2. Failed jobs should be recorded in execution history (with error info)
3. `nextRunAtMs` should correctly update to the next trigger time
4. Optional: Add retry mechanism with backoff
## Workaround
Manually trigger the job to restore scheduling:
```bash
openclaw cron run
```
## Questions
1. Why does `lastStatus` show "ok" when subsequent jobs were not executed?
2. What is the cron scheduler behavior when models are unavailable?
3. Is there a circuit breaker mechanism that prevents future triggers?
4. Are all execution attempts recorded in run history, or only successful ones?
## Steps to Reproduce
1. Configure a cron job (e.g., hourly)
2. Exhaust all configured model quotas (or simulate 429 responses)
3. Wait for the job trigger time
4. Observe whether the job is triggered and if there is an execution record
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.