cloudflare / cloudflare/containers
`inflightRequests` leaks on client-aborted proxied requests — container never stops and bills indefinitely
- Dominant language
- TypeScript
- Stars
- 270
- Forks
- 42
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 4
Description
## Summary
When a client aborts a request while it is being proxied to the container, the `inflightRequests` counter leaks (it is incremented but never decremented). From that moment on, `isActivityExpired()` always returns `false`, so `onActivityExpired()` is never called, the container never receives SIGTERM, and the instance keeps running and billing **indefinitely**. The leak is self-sustaining: the recurring alarm keeps the Durable Object pinned in memory, so the (in-memory) counter is never reset by DO eviction either.
We have hit this twice in production: one instance ran for 17.5 hours (2026-08-09) and one for 79 minutes (2026-08-14, caught early by a billing watchdog) with **zero** traffic after the initial burst, on `sleepAfter = '2m'`.
## Affected versions
Verified in the published source of both `0.2.4` and `0.3.7` (latest at the time of writing). The relevant code is identical in both.
## Mechanism
In `containerFetch` (dist/lib/container.js, `0.3.7` line numbers):
```js
this.inflightRequests++; // L887
try {
this.renewActivityTimeout();
const res = await tcpPort.fetch(containerUrl, request); // L891
...
} catch (e) {
this.decrementInflight(); // L962
...
```
If the **incoming** request is cancelled while `await tcpPort.fetch(...)` is pending (client disconnect/abort — in observability these invocations show `outcome: "canceled"`), the invocation's continuations never run: neither the success path nor the `catch` executes, so `decrementInflight()` is never called. The increment on L887 already happened synchronously. Net effect: `inflightRequests` is permanently ≥ 1.
Then in the alarm path:
```js
isActivityExpired() { // L1687
if (this.inflightRequests > 0) {
this.renewActivityTimeout();
return false;
}
return this.sleepAfterMs <= Date.now();
}
```
Every alarm sees a non-zero counter, renews the deadline by `sleepAfter`, and reschedules. The container is never told to stop, and the alarm cadence itself keeps the DO alive in memory, so the leaked counter never resets. The only way out is external (we used a same-image rolling rollout via the API to force the instance to exit).
## Observed behaviour (production, 2026-08-14)
- 12:20:19Z: burst of ~30 concurrent requests proxied to one container DO during cold start; several show `outcome: "canceled"` in Workers observability.
- 12:20:20Z: last request completes. No traffic to this DO after this point.
- From 12:22 onward: one `alarm` invocation every 120.06s (= `sleepAfter` 2m), `outcome: "ok"`, forever. The `"Activity expired, signalling container to stop"` log line never appears for this DO (sibling DOs from the same deployment that had no cancelled requests logged it and stopped normally).
- Instance state stayed `running` (confirmed via `GET /accounts/{acc}/containers/applications/{app}/instances`) until we forced a rollout at 13:38Z.
## Repro sketch
1. `class MyContainer extends Container { sleepAfter = '2m'; }`, proxy requests via `getContainer(...).fetch(...)`.
2. Fire a request at a cold DO and abort it client-side while the container is still starting (before the proxied fetch resolves).
3. Observe: the DO alarm fires every `sleepAfter` interval indefinitely, `onActivityExpired` is never invoked, and the container instance keeps running (and billing) with zero traffic.
## Suggested direction
`inflightRequests` needs to be robust against invocation cancellation, e.g.:
- decrement from a `request.signal` `abort` listener (if the runtime delivers it in this situation), and/or
- treat the counter as advisory in `isActivityExpired()`: cap how long a non-zero counter may defer expiry without any *new* `renewActivityTimeout()` from real traffic (a leaked counter currently defers it forever), and reconcile/zero it at that bound.
Related but distinct: #162 (alarm firing *during* a long `containerFetch`) — this issue is the mirror image: the alarm never firing the expiry because a cancelled fetch left the counter stuck.
## Workaround
We now run an idle self-exit watchdog inside the container image (exit 0 after N seconds with no requests and no queued work), which bounds the damage regardless of DO-side accounting.
Happy to share account/instance IDs privately for billing-side verification.
Contributor guide
Research direction
Start in dist/lib/container.js at containerFetch, decrementInflight(), and isActivityExpired(), then trace the alarm path and request cancellation behavior described in the issue. Reproduce with a 2-minute sleepAfter and an aborted request during container startup. Done means the counter cannot remain stuck and the alarm eventually invokes onActivityExpired so the idle container stops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100