cloudflare / cloudflare/containers

Container never sleeps after a client abandons a body-bearing response: inflightRequests leaks and sleepAfter renews forever

Open
#242 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
270
Forks
42
Avg merge
1d 5h
Merged PRs (30d)
4

Description

### Version

`@cloudflare/containers` 0.3.7 (latest at time of filing)

### Summary

When a caller abandons a body-bearing response from `containerFetch()` without consuming the body, the inflight-request counter never decrements, `isActivityExpired()` renews the activity timeout on every alarm pass, and the container (plus its supervisor DO) runs — and bills — indefinitely. `sleepAfter` never fires. The only thing that ends the instance is a deploy rolling the image.

### Where

In `containerFetch()` (dist/lib/container.js), the decrement for body-bearing responses is deferred until the body has been fully piped to the consumer:

```js
if (res.body !== null) {
const { readable, writable } = new IdentityTransformStream();
res.body?.pipeTo(writable).finally(() => {
this.decrementInflight();
});
return new Response(readable, res);
}
```

If the returned `readable` is never read (client disconnected, caller timed out and abandoned the await, isolate evicted), `pipeTo` stalls on backpressure forever once the body exceeds the transform stream's buffer, so the `.finally()` never runs.

Meanwhile the activity check treats a pinned counter as endless activity:

```js
isActivityExpired() {
if (this.inflightRequests > 0) {
this.renewActivityTimeout(); // renews forever
return false;
}
return this.sleepAfterMs <= Date.now();
}
```

Small bodies that fit the stream buffer complete the `pipeTo` regardless of a reader, so this only bites once responses exceed the buffer — which is exactly the large-payload case (file transcodes, archives, model output).

### How we hit it

An ffmpeg transcode container returns multi-megabyte mp4 bodies to a Cloudflare Workflow; each step does `const res = await container.fetch(...); await res.arrayBuffer()`. Any step abandoned between those two awaits (step timeout, retry, deploy mid-run) leaks one inflight slot, and that instance never sleeps again despite `sleepAfter = '20m'`.

Over an 11-day window we accumulated ~1,000 idle instance-hours (memory/disk GiB-seconds ÷ instance size) against ~13 hours of actual vCPU work (~98.7% idle). Instances only stopped on deploys. On `standard-3` that idle time was ~$79 of an ~$86 invoice.

### Expected

An abandoned response body should not keep the container awake indefinitely. Some options:

- Tie the pending `pipeTo` to a deadline or to the request's abort signal, so an unread body eventually settles the inflight slot.
- Renew activity only while bytes are actually flowing, rather than unconditionally whenever `inflightRequests > 0`.
- At minimum, document that abandoned reads of body-bearing responses pin the container awake, and recommend an application-level watchdog.

### Workaround

We subclassed `Container` to track last-request time ourselves and used `schedule()` (which the alarm loop processes even in the leaked state) to run a reaper that force-stops (`stop()`, falling back to `destroy()`) after 30 minutes with no new requests.

Contributor guide

Open the contributing guide

Research direction

Start in dist/lib/container.js at containerFetch() and isActivityExpired(), then trace the alarm loop and the body pipe's lifecycle. Reproduce with an unread body larger than the stream buffer; done means an abandoned body no longer keeps inflightRequests active or prevents sleepAfter from stopping the container.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
47/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.