open-telemetry / open-telemetry/opentelemetry-ruby

Asynchronous instruments produce incorrect cumulative and delta output

Open
#2,293 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug stale
Dominant language
Ruby
Stars
606
Forks
301
Avg merge
3d 19h
Merged PRs (30d)
42

Description

Asynchronous instruments (ObservableCounter, ObservableUpDownCounter) produce incorrect output across multiple collections. The callback reports the absolute cumulative value at collection time, but the SDK feeds that value into Sum#update, which does ndp.value += increment (the same increment path synchronous instruments use) and the async stream never differences successive observations. The result is wrong under both temporalities.

With a callback that steadily returns 1000, across three collections:

Temporality Expected Actual
Cumulative 1000, 1000, 1000 1000, 2000, 3000 (absolute value is +='d across collects)
Delta 1000, 0, 0 (first obs is +1000 from start; no change after) 1000, 1000, 1000 (store is cleared each collect, so the raw absolute is re-emitted instead of the difference. A delta consumer that reconstructs a total, e.g. Prometheus total += delta, sums these back to 1000, 2000, 3000)

It's clearest with a monotonic ObservableCounter reporting a running total (e.g. total requests served): a callback steady at 5000 should export 5000, 5000, 5000 (cumulative) or 5000, 0, 0 (delta), but currently gives 5000, 10000, 15000.

Per the Metrics SDK spec, an asynchronous instrument's callback reports the cumulative absolute value; the SDK should export it directly as cumulative, or difference successive observations to derive delta. The current instruments do neither.

Root cause:

  • AsynchronousMetricStream#invoke_callback passes the callback's return value straight into @default_aggregation.update(value, ...). see: asynchronous_metric_stream.rb:62.
  • Sum#update treats its argument as an increment: ndp.value += increment. see: sum.rb:92.

This is correct for synchronous instruments (each .add(x) genuinely is an increment) but incorrect for asynchronous ones (each callback is an absolute total). It has gone unnoticed because the async tests (observable_counter_test.rb, observable_up_down_counter_test.rb) each pull only once, so the across-collect behavior is never exercised.

Share details about your runtime

Operating system details: macOS (Darwin 25)
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.4.9"
RUBY_DESCRIPTION: "ruby 3.4.9 (2026-03-11 revision 76cca827ab) +PRISM [arm64-darwin25]"
opentelemetry-metrics-sdk: 0.16.0
opentelemetry-metrics-api: 0.4.0

Share a simplified reproduction if possible

require 'bundler/inline'

gemfile(true) do
  source 'https://rubygems.org'

  gem 'opentelemetry-metrics-api'
  gem 'opentelemetry-metrics-sdk'
end

require 'opentelemetry-metrics-sdk'

# Minimal exporter that records the value of every data point it sees.
class Capture
  attr_reader :rows

  def initialize = @rows = []

  def export(metrics, timeout: nil)
    metrics.each { |m| m.data_points.each { |dp| @rows << dp.value } }
    OpenTelemetry::SDK::Metrics::Export::SUCCESS
  end

  def force_flush(timeout: nil) = OpenTelemetry::SDK::Metrics::Export::SUCCESS
  def shutdown(timeout: nil) = OpenTelemetry::SDK::Metrics::Export::SUCCESS
end

OpenTelemetry::SDK.configure
cap = Capture.new
reader = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(exporter: cap)
OpenTelemetry.meter_provider.add_metric_reader(reader)

# Callback reports a constant absolute value of 1000 on every collection.
OpenTelemetry.meter_provider.meter('repro')
  .create_observable_up_down_counter('test.oudc', callback: -> { 1000 }, unit: 'By')

3.times { reader.force_flush }

p cap.rows
# Expected: [1000, 1000, 1000] - the observed absolute value each collection.
# Actual:   [1000, 2000, 3000] - the value accumulates across collections.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with metrics_sdk/lib/opentelemetry/sdk/metrics/state/asynchronous_metric_stream.rb:62 and aggregation/sum.rb:92, then compare the existing observable_counter_test.rb and observable_up_down_counter_test.rb coverage. Exercise three collections under cumulative and delta temporalities; done means constant absolute callback values produce 1000, 1000, 1000 cumulatively and 1000, 0, 0 as deltas, with regression tests covering both instruments.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.