getsentry / getsentry/sentry-javascript

Lambda extension stops polling after a 300s invocation, hanging every later invocation on that execution environment

Abierto
#24,218 4 comentarios 0 reacciones 1 asignado Reclamado por @msonnb Ver en GitHub
AWS Lambda Bug javascript
Lenguaje dominante
TypeScript
Estrellas
8.7k
Forks
1.8k
Merge medio
1 d 17 h
PR fusionados (30 d)
523

Descripción

### Which SDK are you using?

`@sentry/aws-serverless`

### SDK Version

10.36.0 (Lambda layer `SentryNodeServerlessSDKv10:49`). Also present on `develop` (10.73.0,
layer `:88`) — the extension source is unchanged, so this is not fixed by upgrading.

### Framework Version

AWS Lambda, `nodejs22.x`

### Steps to Reproduce

No AWS account needed — the official runtime image runs external extensions.

```bash
cd packages/aws-serverless && yarn build:extension

mkdir -p /tmp/repro/task /tmp/repro/opt/extensions /tmp/repro/opt/sentry-extension
cat > /tmp/repro/task/index.js <<'EOF'
exports.handler = async event => {
await new Promise(r => setTimeout(r, Number(event.sleepMs || 0)));
return { ok: true };
};
EOF
cp build/lambda-extension/sentry-extension /tmp/repro/opt/extensions/
cp build/lambda-extension/index.mjs /tmp/repro/opt/sentry-extension/

# AWS_LAMBDA_FUNCTION_TIMEOUT matters: the emulator defaults to 300s, the same number
# under test, and a function timeout at 300s masks the bug entirely.
docker run -d --name repro -p 9000:8080 \
-e AWS_LAMBDA_FUNCTION_TIMEOUT=900 \
-v /tmp/repro/task:/var/task:ro -v /tmp/repro/opt:/opt:ro \
public.ecr.aws/lambda/nodejs:22 index.handler

URL=http://localhost:9000/2015-03-31/functions/function/invocations
curl -s -XPOST $URL -d '{"sleepMs":310000}' # {"ok":true} after ~310s
curl -s --max-time 60 -XPOST $URL -d '{"sleepMs":100}' # never returns
docker logs repro
```

### Expected Result

Both invocations return.

### Actual Result

The second invocation's handler finishes in ~110ms and the response never comes:

```
handler: done
INVOKE RTDONE(status: success, produced bytes: 0, duration: 107.806000ms)
<- nothing after this
```

`External agent sentry-extension ... registered` appears exactly once, so both invocations ran
on the same execution environment — the first one poisoned it. On real Lambda this ends as
`Status: timeout` with the handler long finished, and `PostRuntimeExtensionsDuration` equal to
the full function timeout.

Production numbers from one SQS worker (600s timeout, 155,083 invocations over 7 days):

| statistic | value |
| --- | --- |
| `PostRuntimeExtensionsDuration` p50 / p90 / p99.9 | 0 / 0 / **0.04 ms** |
| maximum | **599,949.55 ms** |

37 invocations hit `Status: timeout` in that window, against 33 that ran past 300s and did not
— about one poisoned environment each, which is what the mechanism predicts.

### Additional Context

**Mechanism.** `/event/next` both acknowledges the previous event and waits for the next one,
so the poll issued after each event stays open for the whole of the following invocation.
Node's `fetch` applies undici's `headersTimeout`, default 300,000 ms — measured against a
server that accepts the connection and never sends headers:
`rejected after 301.0 s -> UND_ERR_HEADERS_TIMEOUT`. Frozen time does not count against it
(undici's clock is tick-driven, `fastNow += TICK_MS`), so only a genuinely long invocation
triggers this, which is why the cliff sits exactly at 300s.

The rejection then escapes the loop, which has no `try`/`catch`:

```js
while (true) { await extension.next(); }
main().catch(err => { DEBUG_BUILD && debug.error('Error in Lambda Extension', err); });
```

**Why it is silent.** Lambda ends an invocation only once the runtime *and every registered
extension* have asked for the next event, so the long invocation itself completes normally and
later ones hang with nothing logged. And `debug` is only enabled from `Sentry.init`, which this
separate process never calls — `debug.error` here cannot print under any option or env var.

**Scope.** The extension is enabled by default for layer users, and `useLayerExtension: false`
does not help: the layer ships the binary in `/opt/extensions/`, which Lambda starts regardless
of SDK options. Reproduced identically with the published `SentryNodeServerlessSDKv10:88` layer,
and does not reproduce with a patched extension.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.