open-telemetry / open-telemetry/opentelemetry-python

PeriodicMetricExporter does not export periodically for either Delta or Cumulative temporality ExponentialHistogram

Open
#3,971 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Describe your environment

OS: Centos 7
Python version: 3.11.1
SDK version: 1.25.0
API version: 1.25.0

What happened?

I have a low-throughput service and have implemented my own Histogram to measure the duration of an operation. This uses a PeriodicMetricExporter which is configured to export every 60 seconds. When I use Delta temporality, I get an export within 60 seconds of the last histogram recording. I then have no exports until the next observation is made.

When changing to Cumulative temporality I have the exact same experience. For comparison purposes, when using the Golang SDK I get the same behaviour as Python's Delta but, when using Cumulative, I get a metronomic export every 60 seconds regardless of any observations being recorded..

Should the Python PeriodicMetricExporter export every 60 seconds or not, and how does temporality setting play into it? At the moment it causes me a problem because my service has low throughput. My observability backend (New Relic) uses the StartTimeUnixNano value which can be hours in the past. A constant periodic export would help to correct this as the StartTimeUnixNano would be at most 60 seconds in the past.

Steps to Reproduce
metrics_temporality = {
        Histogram: AggregationTemporality.DELTA
    }
otlp_metric_exporter = OTLPMetricExporter(endpoint=grpc_collector_address, insecure=True,
                                              preferred_temporality=metrics_temporality, timeout=30_000)
metric_reader = PeriodicExportingMetricReader(otlp_metric_exporter,
                                                  export_interval_millis=60_000))
meter_provider = MeterProvider(metric_readers=[metric_reader], resource=_create_resources(),
                                   views=[View(instrument_type=Histogram,
                                               aggregation=ExponentialBucketHistogramAggregation())])
metrics.set_meter_provider(meter_provider)

meter = metrics.get_meter(__name__)

h = meter.create_histogram(
        name='my_histogram', unit='s', description='just my histogram'
    )

h.record(1.0)
time.sleep(120)
h.record(2.0)
    
Expected Result

I expected to see periodic metric exports for the histogram every 60 seconds, regardless of chosen temporality

Actual Result

A metric export within 60 seconds of the last observation.

Additional context

N/A

Would you like to implement a fix?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with PeriodicExportingMetricReader and the ExponentialBucketHistogramAggregation setup shown in the reproduction, then compare exports for Delta and Cumulative temporality with no observations between recordings. Check the OTLPMetricExporter path and relevant metric-reader behavior. Done means the export cadence and StartTimeUnixNano behavior are defined and covered for both temporality settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.