cloudflare / cloudflare/agents

Bound runFiber recovery by attempts (not only wall-clock age)

Open
#1,728 0 comments 0 reactions 1 assignee Claimed by @threepointone View on GitHub
Dominant language
TypeScript
Stars
5.6k
Forks
711
Avg merge
1d 20h
Merged PRs (30d)
53

Description

## Summary

`runFiber` recovery currently bounds a repeatedly-throwing `onFiberRecovered()` hook by **wall-clock age only** (`fiberRecoveryMaxAgeMs`, default 24h). A deterministically-broken hook is therefore retried for a full day before the orphan row is discarded (`fiber:recovery:skipped` / `max_age_exceeded`), and with `fiberRecoveryMaxAgeMs: 0` ("retain forever") it is retried indefinitely.

A recent change (the recovery-coverage branch) made the recovery follow-up alarm **back off exponentially (capped at 5 min)** on no-progress scans, so this is no longer a busy-loop. But two rough edges remain:

1. **Wall-clock-only bound is coarse.** A transient failure usually clears within a few attempts; a hook that has thrown 5-10 times is almost certainly a permanent bug, not a transient. Grinding it for 24h wastes wakeups and delays any user-visible "we gave up" signal.
2. **`fiberRecoveryMaxAgeMs: 0` keeps the Durable Object warm forever** (it retries on the capped backoff and never idle-evicts while an un-recoverable row exists). This is now documented (see `docs/durable-execution.md` and the `fiberRecoveryMaxAgeMs` JSDoc), but it is still a footgun.

## Proposal

Bound recovery by **attempts** as well as age, mirroring `@cloudflare/think` / `AIChatAgent` chat recovery (which already has `maxAttempts`, default 10):

- Track a durable per-row recovery-attempt count (a column on `cf_agents_runs`, or a small side table).
- Give up after N consecutive failed `onFiberRecovered()` attempts with a clear terminal event (e.g. a new `reason: "max_attempts_exceeded"`, or a dedicated `fiber:recovery:abandoned` event), independent of age.
- For `fiberRecoveryMaxAgeMs: 0`, "retain forever" should then mean "keep the row for inspection but stop self-waking once attempts are exhausted," so the DO can idle-evict. Recovery still resumes opportunistically on the next natural wake.

## Why a separate change

This needs a durable per-row attempt counter (schema/storage change) and changes the semantics of a public option, so it is intentionally scoped out of the starvation-fix + backoff work that surfaced it.

## Acceptance

- A hook that always throws is abandoned after a bounded number of attempts (configurable), with an observable terminal event, regardless of `fiberRecoveryMaxAgeMs`.
- `fiberRecoveryMaxAgeMs: 0` no longer keeps a DO permanently warm for an un-recoverable row.
- Attempt count survives Durable Object restarts (durable, not in-memory).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.