getsentry / getsentry/sentry-javascript

Lambda extension stops polling after a 300s invocation, hanging every later invocation on that execution environment

Aperta
#24,218 4 commenti 0 reazioni 1 assegnatario Rivendicata da @msonnb Vedi su GitHub
AWS Lambda Bug javascript
Lingua principale
TypeScript
Stelle
8.7k
Fork
1.8k
Merge medio
1g 17h
PR unite (30g)
523

Descrizione

### Which SDK are you using?

`@sentry/aws-serverless`

### SDK Version

10.36.0 (Lambda layer `SentryNodeServerlessSDKv10:49`). Also present on `develop` (10.73.0,
layer `:88`) — the extension source is unchanged, so this is not fixed by upgrading.

### Framework Version

AWS Lambda, `nodejs22.x`

### Steps to Reproduce

No AWS account needed — the official runtime image runs external extensions.

```bash
cd packages/aws-serverless && yarn build:extension

mkdir -p /tmp/repro/task /tmp/repro/opt/extensions /tmp/repro/opt/sentry-extension
cat > /tmp/repro/task/index.js <<'EOF'
exports.handler = async event => {
await new Promise(r => setTimeout(r, Number(event.sleepMs || 0)));
return { ok: true };
};
EOF
cp build/lambda-extension/sentry-extension /tmp/repro/opt/extensions/
cp build/lambda-extension/index.mjs /tmp/repro/opt/sentry-extension/

# AWS_LAMBDA_FUNCTION_TIMEOUT matters: the emulator defaults to 300s, the same number
# under test, and a function timeout at 300s masks the bug entirely.
docker run -d --name repro -p 9000:8080 \
-e AWS_LAMBDA_FUNCTION_TIMEOUT=900 \
-v /tmp/repro/task:/var/task:ro -v /tmp/repro/opt:/opt:ro \
public.ecr.aws/lambda/nodejs:22 index.handler

URL=http://localhost:9000/2015-03-31/functions/function/invocations
curl -s -XPOST $URL -d '{"sleepMs":310000}' # {"ok":true} after ~310s
curl -s --max-time 60 -XPOST $URL -d '{"sleepMs":100}' # never returns
docker logs repro
```

### Expected Result

Both invocations return.

### Actual Result

The second invocation's handler finishes in ~110ms and the response never comes:

```
handler: done
INVOKE RTDONE(status: success, produced bytes: 0, duration: 107.806000ms)
<- nothing after this
```

`External agent sentry-extension ... registered` appears exactly once, so both invocations ran
on the same execution environment — the first one poisoned it. On real Lambda this ends as
`Status: timeout` with the handler long finished, and `PostRuntimeExtensionsDuration` equal to
the full function timeout.

Production numbers from one SQS worker (600s timeout, 155,083 invocations over 7 days):

| statistic | value |
| --- | --- |
| `PostRuntimeExtensionsDuration` p50 / p90 / p99.9 | 0 / 0 / **0.04 ms** |
| maximum | **599,949.55 ms** |

37 invocations hit `Status: timeout` in that window, against 33 that ran past 300s and did not
— about one poisoned environment each, which is what the mechanism predicts.

### Additional Context

**Mechanism.** `/event/next` both acknowledges the previous event and waits for the next one,
so the poll issued after each event stays open for the whole of the following invocation.
Node's `fetch` applies undici's `headersTimeout`, default 300,000 ms — measured against a
server that accepts the connection and never sends headers:
`rejected after 301.0 s -> UND_ERR_HEADERS_TIMEOUT`. Frozen time does not count against it
(undici's clock is tick-driven, `fastNow += TICK_MS`), so only a genuinely long invocation
triggers this, which is why the cliff sits exactly at 300s.

The rejection then escapes the loop, which has no `try`/`catch`:

```js
while (true) { await extension.next(); }
main().catch(err => { DEBUG_BUILD && debug.error('Error in Lambda Extension', err); });
```

**Why it is silent.** Lambda ends an invocation only once the runtime *and every registered
extension* have asked for the next event, so the long invocation itself completes normally and
later ones hang with nothing logged. And `debug` is only enabled from `Sentry.init`, which this
separate process never calls — `debug.error` here cannot print under any option or env var.

**Scope.** The extension is enabled by default for layer users, and `useLayerExtension: false`
does not help: the layer ships the binary in `/opt/extensions/`, which Lambda starts regardless
of SDK options. Reproduced identically with the published `SentryNodeServerlessSDKv10:88` layer,
and does not reproduce with a patched extension.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.