open-telemetry / open-telemetry/opentelemetry-python-contrib
Provide alternate strategies for exporting metrics with AWS Lambda
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 1.1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 16
Description
What problem do you want to solve?
The current AWS Lambda instrumentation library flushes all metrics after each invocation, which results in metric data-points being generated after each invocation, regardless of the configured export interval. This issue becomes increasingly problematic for Lambda functions with short execution durations (e.g. ~100ms) since even datapoints with the same attributes, will not have the ability to be aggregated if they span more than a single invocation. In addition to being operationally inefficient, because almost all observability vendors charge based on the number of datapoints ingested, this pattern results in dramatic increases in costs.
Describe the solution you'd like
There are several changes which can be employed to help reduce the number of metric datapoints being generated. The two initial changes are as follows:
- Provide a configuration option/environment variable to disable force flushing metrics after each invocation.
- Provide a configuration option/environment variable to register a signal handler for the
SIGTERMshutdown event to flush all metrics before the Lambda instance is cleaned up. (see https://aws.amazon.com/blogs/compute/python-3-12-runtime-now-available-in-aws-lambda/) - Either: add an enhancement to the existing PeriodicExportingMetricReader to add the ability to suspend/resume metric collection to ensure that metric collection stops before the Lambda environment is potentially frozen OR create a new
SuspendableExportingMetricReaderwhich has the same functionality.
Describe alternatives you've considered
Metric aggregation in the collector, however there seems to be limited support for this pattern.
Additional Context
No response
Would you like to implement a fix?
Yes
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the AWS Lambda instrumentation and tracing how it flushes metrics after each invocation, then read the PeriodicExportingMetricReader behavior and the SIGTERM shutdown context. Done means the requested export strategies are defined, their configuration and lifecycle behavior are specified, and the resulting metric-flush behavior is covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100