open-telemetry / open-telemetry/opentelemetry-lambda
Coldstart Processor creating broken traces for cold start and next invocation
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 432
- Forks
- 246
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 47
Description
Describe the bug
I was unable to get the coldstart processor to do what is says that it does on the tin. If i submit 4 requests to a helloworld function behind an api gateway I only end up with 3 traces in my observability platform. Two of these traces will use the same internal spans, have missing spans and appears to be related to the cold start. One trace goes missing. Invocations from coldstart + 3 appear normal.
Steps to reproduce
- Create a hello world Lambda function fronted by either a REST or HTTP api in AWS
- Deploy the api and function with the otel collector and nodejs layers bundled with a custom collector.yaml that enables the coldstart processor. ie.
processors:
coldstart:
resource:
attributes:
- key: deployment.environment.name
value: '${env:ENVIRONMENT_NAME}'
action: upsert
Additional configure the collector.yaml to send to your observability platform of choice.
I had the following env variables set for the function:
// OpenTelemetry configuration
AWS_LAMBDA_EXEC_WRAPPER: '/opt/otel-handler',
OPENTELEMETRY_COLLECTOR_CONFIG_URI: '/var/task/collector.yaml',
OTEL_TRACES_SAMPLER: 'always_on',
OTEL_PROPAGATORS: 'tracecontext,baggage',
OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION: 'true',
OTEL_NODE_ENABLED_INSTRUMENTATIONS: 'aws-sdk,aws-lambda,http,undici',
OTEL_SERVICE_NAME: serviceName,
OPENTELEMETRY_EXTENSION_LOG_LEVEL: 'INFO',
I tried active tracing on, passthrough and off.
I tried the insights layer on and off.
- Manually trigger a http get request to the api/function endpoint four times
- Check traces that arrive in the observability platform
What did you expect to see?
I expected to see 4 traces arrive in my observability platform (grafana cloud). I expected the first trace that would have triggered a cold start to have a trace that includes a span for the init phase and be of a longer duration than the other three.
What did you see instead?
- Only three traces arrived.
-
One trace went missing
-
The first two traces are broken
- trace 1
- trace 2
- trace 3
What version of collector/language SDK version did you use?
- 0.19.0 (collector layer)
- 0.19.0 (nodejs layer)
What language layer did you use?
nodejs
Additional context
I tried a few different things to try and get this to work:
- bundling to commonjs and esm
- using a http or rest api gateway
- removing the batch and decouple processors
- putting the coldstart processor first/last in order of execution
- setting the function active tracing to on, passthrough and off.
- removing the insight lambda layer
Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the four-request scenario with the 0.19.0 Node.js and collector layers, using the shown collector.yaml coldstart configuration. Compare the resulting traces and collector output, including the first invocation's init span; done means four distinct complete traces arrive and the cold-start trace includes the expected initialization data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, nodejs
- Domain
- cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100