open-telemetry / open-telemetry/opentelemetry-lambda
Telemetryapi receiver slow startup time (~3.5-4 seconds)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 432
- Forks
- 246
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 47
Description
Describe the bug
With a hello world nodejs lambda function with 512Mb of memory Im seeing the telemetry api receiver take over 3.5 seconds to initialize during the function cold start init phase.
Steps to reproduce
- Create a hello world nodejs function
- Add the opentelemtry-lambda collector and nodejs layers
- Add a custom colllectory.yaml to the function bundling step that enables the telemetryapi receiver ie
receivers:
otlp:
protocols:
grpc:
endpoint: 'localhost:4317'
http:
endpoint: 'localhost:4318'
# This receiver is used to capture the logs sent to stdout by the Lambda function
# This is how we capture logs generated by the Powertools Logger.
telemetryapi:
Also add the receiver to a valid logs pipeline to capture the telemetry from the receiver
- Deploy the function behind and api gateway
- Make some calls to the endpoint
What did you expect to see?
Functionally everything is working, the telemetry api receiver is a big win. I was expecting this to startup in less than a second though.
What did you see instead?
The init phase for the telemetry receiver is consistently over three seconds:
What version of collector/language SDK version did you use?
- 0.19.0 (collector layer)
- 0.19.0 (nodejs layer)
What language layer did you use?
nodejs
Additional context
With the telemetryapi receiver turned on Im seeing overall coldstart init times of ~4+ seconds. With an instrumented custom authorizer function on the gateway I'm seeing a worst case scenario for a hello world function of ~10.5 second round trip time for a cold start (4.5 seconds for the auth function to spin up and then 4.5 seconds for the helloworld function to spin up).
My serverless mircoservice architecture can tolerate the occasional blow out in a cold start time but when cold starts are consistently this long it means that hitting my stack when cold causes service to service interactions to quickly hit the function timeouts (20-30 seconds) and exceeds user tolerable page load times.
I like/need the telemetryapi receiver functionality. I would like the function cold start init phase telemetry it provides shipped to my non cloudwatch observability stack. My code is currently using a logging framework that emits to stdout and the telemtryapi receiver allows me to capture this and ship it to somewhere other than cloudwatch.
For my usecase I can't use the telemetryapi receiver with the current start times without resorting to provisioned concurrency/managed instances which are imperfect solutions.
Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the hello-world Node.js Lambda setup with the collector and Node.js layers, enabling the telemetryapi receiver as described. Compare cold-start initialization with and without the receiver, then inspect the collector and layer startup paths to identify the delay. Done means reducing startup time substantially while preserving telemetry capture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, nodejs
- Domain
- backend, cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100