aws / aws/aws-lambda-web-adapter

Freeze/thaw poisons pooled localhost connection on non-SnapStart functions → SendRequest/ConnectionReset with no retry; no config to disable pooling

Open
#830 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.7k
Forks
161
Avg merge
2d 10h
Merged PRs (30d)
2

Description

### Summary

On a regular (non-SnapStart) on-demand function, the execution environment freeze/thaw between invocations can leave the adapter's pooled keep-alive connection to the app server in a reset state. The next invocation reuses that dead connection, and because `fetch_response` forwards the request exactly once with no retry, the caller gets a `5xx`. There is currently no configuration to disable the adapter↔app connection pool outside of SnapStart.

### Setup

- Adapter: `public.ecr.aws/awsguru/aws-lambda-adapter:1.0.1` (x86_64), as a Lambda extension in a container image
- App: Next.js 16 standalone server on `:3000`
- Base image: `node:24-bookworm-slim`
- Front: API Gateway v2, payload format 2.0, `$default` route
- Region: `eu-central-1`
- Traffic: low, function scales to zero frequently
- `AWS_LWA_READINESS_CHECK_PATH=/api/health`, `AWS_LWA_READINESS_CHECK_PROTOCOL=tcp`

### Symptom

Intermittent `500 {"message":"Internal Server Error"}` to the client. Adapter log on the failing invocation:

```
ERROR Lambda runtime invoke{requestId="7ab2a21b-..."}:
hyper_util::client::legacy::Error(SendRequest,
hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))
```

The next request to the same execution environment succeeds. The app server does **not** crash — no error on its side, and it keeps serving.

### Timeline (one execution environment, from CloudWatch)

```
15:49:43.137 new exec env starts: app "Ready", "lambda-adapter State: Ready"
15:49:43.141 req A durationMs 760 initDurationMs 562 status success <- cold-start request, OK
15:49:52.105 req B durationMs 85 (warm, no init) status success <- reuses pooled conn, OK
15:49:55.823 req C durationMs 2486 (warm, no init) ERROR ConnectionReset <- 500 to caller
```

Req C is ~3.6s after req B, on a warm environment. The environment was frozen in between.

### Analysis

The adapter builds its client with a keep-alive pool:

https://github.com/aws/aws-lambda-web-adapter/blob/v1.0.1/src/lib.rs#L588-L600

```rust
if env::var("AWS_LAMBDA_INITIALIZATION_TYPE").as_deref() == Ok("snap-start") {
builder.pool_max_idle_per_host(0);
} else {
builder.pool_idle_timeout(Duration::from_secs(4));
}
```

`pool_idle_timeout` is measured with `Instant`/`CLOCK_MONOTONIC`, which does not advance while the execution environment is frozen. So a connection that has been idle far longer than 4s in wall-clock time still looks fresh to the pool and is reused — but the underlying socket was reset across the freeze/thaw. `fetch_response` then does:

https://github.com/aws/aws-lambda-web-adapter/blob/v1.0.1/src/lib.rs#L1012

```rust
let mut app_response = self.client.request(request).await?;
```

— a single attempt. The connection error propagates straight out as a runtime error → `5xx`.

This is the same failure mode the SnapStart branch above guards against with `pool_max_idle_per_host(0)`, but on-demand environments freeze/thaw as well and hit it too. Related prior reports of the `SendRequest` / `IncompleteMessage` family: #415, #295.

### Request

Either (ideally both):

1. **Expose the pool control as configuration** — e.g. `AWS_LWA_DISABLE_CONNECTION_POOL=true` (or `AWS_LWA_POOL_MAX_IDLE_PER_HOST=0`) so non-SnapStart functions can opt into `pool_max_idle_per_host(0)`. For localhost the cost of a fresh connection per request is negligible.
2. **Retry the forwarded request once on connection-level errors** (`ConnectionReset`, `BrokenPipe`, `IncompleteMessage` when zero bytes were written). These are safe to retry — no request bytes reached the app — and every mainstream HTTP client does this for pooled connections.

### Workarounds tried

- `AWS_LWA_READINESS_CHECK_PROTOCOL=tcp` — removes the connection the readiness probe would otherwise seed into the pool, but not request-to-request reuse (this report is with `tcp` already set).
- Client-side retry at the caller — works for service-to-service callers, not for browsers hitting the function directly.

Contributor guide

Open the contributing guide

Research direction

Start in src/lib.rs at the client builder around lines 588-600 and fetch_response around line 1012, then trace how pooled requests surface connection errors. Review the existing client configuration and forwarding tests, if present. Done means the freeze/thaw failure is addressed through the requested pool control or safe connection-error retry, with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, cloud, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.