open-telemetry / open-telemetry/opentelemetry-python

Histogram metrics are much larger in v1.23.0

Open
#3,959 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

We noticed a very large increase in our GCM cost due to an increase in metrics bytes ingested for our base histogram metrics (e.g. http.client.duration). This coincided with an upgrade to v1.23.0. A subsequent downgrade to v1.22.0 led to a decrease in the bytes ingested and cost increases back to their prior levels.

Screenshot 2024-06-06 at 5 49 44 PM

This commit is the revert: https://github.com/Recidiviz/pulse-data/commit/d321a4e30f612e9964f18106ded28d6a0fce250e

Steps to reproduce
Describe exactly how to reproduce the error. Include a code sample if applicable.

Upgrade to v1.23.0 or later (only tested up to v1.24.0, so it is possible it has been fixed)

What is the expected behavior?

No increase in bytes ingested by GCM for histogram metrics.

What is the actual behavior?

Order of magnitude increase in cost.

Additional context

I haven't taken the time to fully understand the changes here, but if this PR led to all of the buckets always being created, and before that was not the case, this could be the culprit: https://github.com/open-telemetry/opentelemetry-python/pull/3429

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading PR #3429 and the linked revert commit d321a4e30f612e9964f18106ded28d6a0fce250e, then compare histogram behavior between v1.22.0 and v1.23.0. Verify whether all buckets are created and whether the change explains the increased GCM-ingested bytes; done means the regression is reproduced, explained, and validated against a fixed version or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.