aws / aws/aws-xray-sdk-node

Lambda mode shares one CLS context and facade segment across concurrent invocations (Lambda Managed Instances)

Open
#760 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
280
Forks
157
PR merge metrics
No merged PRs in 30d

Description

### Summary

In Lambda mode, the SDK creates a single process-wide CLS context at initialization and never creates another (`lib/env/aws_lambda.js`):

```js
var namespace = contextUtils.getNamespace();
namespace.enter(namespace.createContext());
contextUtils.setSegment(facadeSegment());
```

With [Lambda Managed Instances](https://docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances.html) (`perExecutionEnvironmentMaxConcurrency` > 1), multiple invocations are multiplexed into the same execution environment concurrently. Two pieces of state are then shared by all in-flight invocations:

1. **The active-segment slot.** `setSegment()` writes into the single shared context, so any code that manages subsegments via `getSegment()`/`setSegment()` (rather than `captureAsyncFunc`) resolves another invocation's subsegment once invocations interleave. Annotations, metadata, and instrumented-call subsegments attach to the wrong invocation - potentially another tenant's trace.

2. **The facade segment itself.** There is one facade per process, and `resolveLambdaTraceData()` mutates its `trace_id`/`id`/`notTraced` in place whenever `getSegment()` observes a new trace header. Under interleaving, subsegment `flush()` reads `this.segment.trace_id` *at emission time*, so even `captureAsyncFunc` users can emit subsegments onto whichever trace the facade was last resolved to.

The runtime already provides the primitive needed to fix this: `globalThis.awslambda.InvokeStore`, an `AsyncLocalStorage` that the runtime scopes around each invocation - and which `resolveLambdaTraceData()` already consults for the trace id.

### Proposal

In Lambda mode, when `awslambda.InvokeStore` is available:

- scope the current-segment storage per invocation (e.g. key the segment slot by `InvokeStore` identity instead of the single CLS slot), and
- maintain a per-invocation facade (or at minimum snapshot the resolved trace data onto subsegments at creation time rather than reading the shared facade at flush time),

falling back to today's single-context behavior on runtimes without `InvokeStore`. This would make both `captureAsyncFunc` and `getSegment()`/`setSegment()`-based instrumentation safe under LMI concurrency without API changes.

### Reproduction

Deploy any function using `getSegment()`/`addNewSubsegment()`/`setSegment()` per invocation on a capacity provider with `perExecutionEnvironmentMaxConcurrency` > 1, add a distinctive annotation per invocation, and force multiplexing: annotations land on other invocations' subsegments, and subsegment documents are emitted with the wrong `trace_id`.

Downstream report with full trace evidence: aws-powertools/powertools-lambda-typescript#5434.

Contributor guide

Open the contributing guide

Research direction

Start in lib/env/aws_lambda.js and trace contextUtils.getNamespace(), setSegment(), getSegment(), and resolveLambdaTraceData(). Reproduce concurrent invocations with perExecutionEnvironmentMaxConcurrency > 1, then verify that segment state and emitted subsegments remain associated with each invocation and trace, while runtimes without awslambda.InvokeStore retain current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript, node.js
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.