cloudflare / cloudflare/containers
Container reported not-running / crashed while genuinely alive during slow startup, even with an immediate placeholder port
- Dominant language
- TypeScript
- Stars
- 270
- Forks
- 42
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 4
Description
## Summary
Running a container whose entrypoint takes anywhere from several seconds to tens of minutes before binding its configured port, we consistently see the container torn down / reported as not-running while it is still genuinely alive and making progress — even after adding a placeholder listener that answers the port-ready health check (`waitForPort()`'s `tcpPort.fetch('http://ping')`) immediately. Across many real attempts we've seen at least three different error signatures, and log evidence of overlapping/duplicate container start attempts within the same short window.
## Environment
- `@cloudflare/containers`: 0.3.7
- `wrangler`: 4.107.0
- Instance type: `standard-2` (1 vCPU / 6GiB / 12GB disk)
- `max_instances: 1`
- Container image: Debian-based (`nextcloud:apache`), PHP/Apache workload with a slow first-boot install step (occ maintenance:install against a remote Postgres DB, ~20-50 min end-to-end) and a separate fast (~seconds) config-restore path used on subsequent boots.
## What we're seeing
### Error 1: `Container crashed while checking for ports, did you start the container and setup the entrypoint correctly?`
This fires from `waitForPort()`'s retry loop (`src/lib/container.ts`) whenever `this.container.running` reads `false` on any poll iteration — regardless of our own configured `portReadyTimeoutMS` (we set this to 3600000ms specifically to tolerate a long install). We confirmed via direct database inspection that the underlying process was still alive and making real progress (creating schema, inserting rows) when this fired.
### Error 2 (after adding a placeholder listener): `The container is not running, consider calling start()`
This string doesn't appear anywhere in the `@cloudflare/containers` package source — it must come from the native `getTcpPort().fetch()` binding itself. We saw this thrown from `containerFetch()`'s proxy call **immediately after** our own explicit `startAndWaitForPorts()` call had just returned successfully (same-second timestamps in `wrangler tail`), and — in the same test — the container's *own* outbound callback to our Worker (proving it was alive) succeeded ~18 seconds *after* this "not running" error was reported.
### Error 3: `There is no container instance that can be provided to this Durable Object, try again later`
Seen in a later attempt, in the same test run as a duplicate/unexpected second incoming request to the same endpoint we only called once, and an `onStop` callback firing with a non-zero exit code — suggestive of more than one container start attempt overlapping for the same Durable Object within a ~30 second window.
## The placeholder-listener workaround (partial)
Since `waitForPort()` accepts *any* valid HTTP response (not just 2xx) as "ready," we tried opening a trivial placeholder HTTP server (`php -S`) on the target port immediately, before any of our slow startup work, specifically to make the readiness check pass on its first poll instead of spending the whole slow-startup window looking unready. This did change the failure signature (we stopped seeing Error 1 as consistently) but did **not** eliminate the underlying issue — we still hit Errors 2 and 3 above on subsequent attempts, including cases where the container appeared to serve a request against a filesystem that hadn't finished syncing yet, alongside signs of overlapping start attempts.
## What we'd like to understand
1. Is there a platform-level (not `portReadyTimeoutMS`/`instanceGetTimeoutMS`) timeout or liveness-check interval that can independently mark a container as not-running while its main process is still alive?
2. Is `this.container.running` guaranteed to be a real-time, non-stale reflection of platform state at the moment it's read, or can it lag/flap? (We found cloudflare/workerd#4362 describing a related staleness issue with this exact flag, in the opposite direction — reporting `true` when it should be `false`.)
3. Are concurrent/overlapping calls into `startAndWaitForPorts()` / `containerFetch()` for the same Durable Object instance expected to be safe, or is there a known race when multiple requests each try to ensure the container is started around the same time?
4. Is the placeholder-listener pattern (open a cheap port immediately, do slow work, then hand off to the real listener) a supported/sane approach for long-starting workloads, or is there a better-supported mechanism we're missing for this class of workload?
Happy to share full `wrangler tail` logs, our Dockerfile, and Worker code if useful — trimmed the above for length and to remove app-specific identifiers, but the mechanism and error strings are copied verbatim from real runs.
Related, possibly relevant issues we found while investigating: #162, #147.
Contributor guide
Assessment
This issue has not been assessed yet.