open-telemetry / open-telemetry/opentelemetry-python-contrib

Provide alternate strategies for exporting metrics with AWS Lambda

Open
#4,157 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

What problem do you want to solve?

The current AWS Lambda instrumentation library flushes all metrics after each invocation, which results in metric data-points being generated after each invocation, regardless of the configured export interval. This issue becomes increasingly problematic for Lambda functions with short execution durations (e.g. ~100ms) since even datapoints with the same attributes, will not have the ability to be aggregated if they span more than a single invocation. In addition to being operationally inefficient, because almost all observability vendors charge based on the number of datapoints ingested, this pattern results in dramatic increases in costs.

Describe the solution you'd like

There are several changes which can be employed to help reduce the number of metric datapoints being generated. The two initial changes are as follows:

  1. Provide a configuration option/environment variable to disable force flushing metrics after each invocation.
  2. Provide a configuration option/environment variable to register a signal handler for the SIGTERM shutdown event to flush all metrics before the Lambda instance is cleaned up. (see https://aws.amazon.com/blogs/compute/python-3-12-runtime-now-available-in-aws-lambda/)
  3. Either: add an enhancement to the existing PeriodicExportingMetricReader to add the ability to suspend/resume metric collection to ensure that metric collection stops before the Lambda environment is potentially frozen OR create a new SuspendableExportingMetricReader which has the same functionality.
Describe alternatives you've considered

Metric aggregation in the collector, however there seems to be limited support for this pattern.

Additional Context

No response

Would you like to implement a fix?

Yes

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the AWS Lambda instrumentation and tracing how it flushes metrics after each invocation, then read the PeriodicExportingMetricReader behavior and the SIGTERM shutdown context. Done means the requested export strategies are defined, their configuration and lifecycle behavior are specified, and the resulting metric-flush behavior is covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.