cloudflare / cloudflare/containers
Container never sleeps after a client abandons a body-bearing response: inflightRequests leaks and sleepAfter renews forever
- Dominant language
- TypeScript
- Stars
- 270
- Forks
- 42
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 4
Description
### Version
`@cloudflare/containers` 0.3.7 (latest at time of filing)
### Summary
When a caller abandons a body-bearing response from `containerFetch()` without consuming the body, the inflight-request counter never decrements, `isActivityExpired()` renews the activity timeout on every alarm pass, and the container (plus its supervisor DO) runs — and bills — indefinitely. `sleepAfter` never fires. The only thing that ends the instance is a deploy rolling the image.
### Where
In `containerFetch()` (dist/lib/container.js), the decrement for body-bearing responses is deferred until the body has been fully piped to the consumer:
```js
if (res.body !== null) {
const { readable, writable } = new IdentityTransformStream();
res.body?.pipeTo(writable).finally(() => {
this.decrementInflight();
});
return new Response(readable, res);
}
```
If the returned `readable` is never read (client disconnected, caller timed out and abandoned the await, isolate evicted), `pipeTo` stalls on backpressure forever once the body exceeds the transform stream's buffer, so the `.finally()` never runs.
Meanwhile the activity check treats a pinned counter as endless activity:
```js
isActivityExpired() {
if (this.inflightRequests > 0) {
this.renewActivityTimeout(); // renews forever
return false;
}
return this.sleepAfterMs <= Date.now();
}
```
Small bodies that fit the stream buffer complete the `pipeTo` regardless of a reader, so this only bites once responses exceed the buffer — which is exactly the large-payload case (file transcodes, archives, model output).
### How we hit it
An ffmpeg transcode container returns multi-megabyte mp4 bodies to a Cloudflare Workflow; each step does `const res = await container.fetch(...); await res.arrayBuffer()`. Any step abandoned between those two awaits (step timeout, retry, deploy mid-run) leaks one inflight slot, and that instance never sleeps again despite `sleepAfter = '20m'`.
Over an 11-day window we accumulated ~1,000 idle instance-hours (memory/disk GiB-seconds ÷ instance size) against ~13 hours of actual vCPU work (~98.7% idle). Instances only stopped on deploys. On `standard-3` that idle time was ~$79 of an ~$86 invoice.
### Expected
An abandoned response body should not keep the container awake indefinitely. Some options:
- Tie the pending `pipeTo` to a deadline or to the request's abort signal, so an unread body eventually settles the inflight slot.
- Renew activity only while bytes are actually flowing, rather than unconditionally whenever `inflightRequests > 0`.
- At minimum, document that abandoned reads of body-bearing responses pin the container awake, and recommend an application-level watchdog.
### Workaround
We subclassed `Container` to track last-request time ourselves and used `schedule()` (which the alarm loop processes even in the leaked state) to run a reaper that force-stops (`stop()`, falling back to `destroy()`) after 30 minutes with no new requests.
Contributor guide
Research direction
Start in dist/lib/container.js at containerFetch() and isActivityExpired(), then trace the alarm loop and the body pipe's lifecycle. Reproduce with an unread body larger than the stream buffer; done means an abandoned body no longer keeps inflightRequests active or prevents sleepAfter from stopping the container.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 47/100