aws / aws/aws-lambda-web-adapter
Freeze/thaw poisons pooled localhost connection on non-SnapStart functions → SendRequest/ConnectionReset with no retry; no config to disable pooling
- Dominant language
- Rust
- Stars
- 2.7k
- Forks
- 161
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 2
Description
### Summary
On a regular (non-SnapStart) on-demand function, the execution environment freeze/thaw between invocations can leave the adapter's pooled keep-alive connection to the app server in a reset state. The next invocation reuses that dead connection, and because `fetch_response` forwards the request exactly once with no retry, the caller gets a `5xx`. There is currently no configuration to disable the adapter↔app connection pool outside of SnapStart.
### Setup
- Adapter: `public.ecr.aws/awsguru/aws-lambda-adapter:1.0.1` (x86_64), as a Lambda extension in a container image
- App: Next.js 16 standalone server on `:3000`
- Base image: `node:24-bookworm-slim`
- Front: API Gateway v2, payload format 2.0, `$default` route
- Region: `eu-central-1`
- Traffic: low, function scales to zero frequently
- `AWS_LWA_READINESS_CHECK_PATH=/api/health`, `AWS_LWA_READINESS_CHECK_PROTOCOL=tcp`
### Symptom
Intermittent `500 {"message":"Internal Server Error"}` to the client. Adapter log on the failing invocation:
```
ERROR Lambda runtime invoke{requestId="7ab2a21b-..."}:
hyper_util::client::legacy::Error(SendRequest,
hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))
```
The next request to the same execution environment succeeds. The app server does **not** crash — no error on its side, and it keeps serving.
### Timeline (one execution environment, from CloudWatch)
```
15:49:43.137 new exec env starts: app "Ready", "lambda-adapter State: Ready"
15:49:43.141 req A durationMs 760 initDurationMs 562 status success <- cold-start request, OK
15:49:52.105 req B durationMs 85 (warm, no init) status success <- reuses pooled conn, OK
15:49:55.823 req C durationMs 2486 (warm, no init) ERROR ConnectionReset <- 500 to caller
```
Req C is ~3.6s after req B, on a warm environment. The environment was frozen in between.
### Analysis
The adapter builds its client with a keep-alive pool:
https://github.com/aws/aws-lambda-web-adapter/blob/v1.0.1/src/lib.rs#L588-L600
```rust
if env::var("AWS_LAMBDA_INITIALIZATION_TYPE").as_deref() == Ok("snap-start") {
builder.pool_max_idle_per_host(0);
} else {
builder.pool_idle_timeout(Duration::from_secs(4));
}
```
`pool_idle_timeout` is measured with `Instant`/`CLOCK_MONOTONIC`, which does not advance while the execution environment is frozen. So a connection that has been idle far longer than 4s in wall-clock time still looks fresh to the pool and is reused — but the underlying socket was reset across the freeze/thaw. `fetch_response` then does:
https://github.com/aws/aws-lambda-web-adapter/blob/v1.0.1/src/lib.rs#L1012
```rust
let mut app_response = self.client.request(request).await?;
```
— a single attempt. The connection error propagates straight out as a runtime error → `5xx`.
This is the same failure mode the SnapStart branch above guards against with `pool_max_idle_per_host(0)`, but on-demand environments freeze/thaw as well and hit it too. Related prior reports of the `SendRequest` / `IncompleteMessage` family: #415, #295.
### Request
Either (ideally both):
1. **Expose the pool control as configuration** — e.g. `AWS_LWA_DISABLE_CONNECTION_POOL=true` (or `AWS_LWA_POOL_MAX_IDLE_PER_HOST=0`) so non-SnapStart functions can opt into `pool_max_idle_per_host(0)`. For localhost the cost of a fresh connection per request is negligible.
2. **Retry the forwarded request once on connection-level errors** (`ConnectionReset`, `BrokenPipe`, `IncompleteMessage` when zero bytes were written). These are safe to retry — no request bytes reached the app — and every mainstream HTTP client does this for pooled connections.
### Workarounds tried
- `AWS_LWA_READINESS_CHECK_PROTOCOL=tcp` — removes the connection the readiness probe would otherwise seed into the pool, but not request-to-request reuse (this report is with `tcp` already set).
- Client-side retry at the caller — works for service-to-service callers, not for browsers hitting the function directly.
Contributor guide
Research direction
Start in src/lib.rs at the client builder around lines 588-600 and fetch_response around line 1012, then trace how pooled requests surface connection errors. Review the existing client configuration and forwarding tests, if present. Done means the freeze/thaw failure is addressed through the requested pool control or safe connection-error retry, with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- backend, cloud, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100