prometheus-community / prometheus-community/stackdriver_exporter

Duplicate metrics from Stackdriver for stackdriver_pubsub_topic_pubsub_googleapis_com_topic metrics

Open
#166 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
117
Avg merge
9d 14h
Merged PRs (30d)
7

Description

For a while our prometheus was not able to scrape any metrics from the stackdriver_export with the following error:

[from Gatherer #2] collected metric "stackdriver_pubsub_topic_pubsub_googleapis_com_topic_send_request_latencies" { label:<name:"has_ordering" value:"false" > label:<name:"project_id" value:"" > label:<name:"response_code" value:"success" > label:<name:"schema_type" value:"none" > label:<name:"topic_id" value:"re.adx_valuation_sampling_requests.v1" > label:<name:"unit" value:"us" > histogram:<sample_count:4206 sample_sum:1.1320835231589793e+08 bucket:<cumulative_count:0 upper_bound:1.19209 > bucket:<cumulative_count:0 upper_bound:1.4305080000000001 > bucket:<cumulative_count:0 upper_bound:1.7166096000000002 > bucket:<cumulative_count:0 upper_bound:2.05993152 > bucket:<cumulative_count:0 upper_bound:2.471917824 > bucket:<cumulative_count:0 upper_bound:2.9663013888000003 > bucket:<cumulative_count:0 upper_bound:3.55956166656 > bucket:<cumulative_count:0 upper_bound:4.271473999872 > bucket:<cumulative_count:0 upper_bound:5.1257687998464 > bucket:<cumulative_count:0 upper_bound:6.15092255981568 > bucket:<cumulative_count:0 upper_bound:7.381107071778816 > bucket:<cumulative_count:0 upper_bound:8.857328486134579 > bucket:<cumulative_count:0 upper_bound:10.628794183361494 > bucket:<cumulative_count:0 upper_bound:12.754553020033793 > bucket:<cumulative_count:0 upper_bound:15.30546362404055 > bucket:<cumulative_count:0 upper_bound:18.36655634884866 > bucket:<cumulative_count:0 upper_bound:22.03986761861839 > bucket:<cumulative_count:0 upper_bound:26.447841142342067 > bucket:<cumulative_count:0 upper_bound:31.737409370810482 > bucket:<cumulative_count:0 upper_bound:38.084891244972574 > bucket:<cumulative_count:0 upper_bound:45.701869493967095 > bucket:<cumulative_count:0 upper_bound:54.84224339276051 > bucket:<cumulative_count:0 upper_bound:65.8106920713126 > bucket:<cumulative_count:0 upper_bound:78.97283048557513 > bucket:<cumulative_count:0 upper_bound:94.76739658269014 > bucket:<cumulative_count:0 upper_bound:113.72087589922819 > bucket:<cumulative_count:0 upper_bound:136.46505107907382 > bucket:<cumulative_count:0 upper_bound:163.7580612948886 > bucket:<cumulative_count:0 upper_bound:196.5096735538663 > bucket:<cumulative_count:0 upper_bound:235.81160826463955 > bucket:<cumulative_count:0 upper_bound:282.97392991756743 > bucket:<cumulative_count:0 upper_bound:339.5687159010809 > bucket:<cumulative_count:0 upper_bound:407.482459081297 > bucket:<cumulative_count:0 upper_bound:488.9789508975564 > bucket:<cumulative_count:0 upper_bound:586.7747410770677 > bucket:<cumulative_count:0 upper_bound:704.1296892924812 > bucket:<cumulative_count:0 upper_bound:844.9556271509774 > bucket:<cumulative_count:0 upper_bound:1013.946752581173 > bucket:<cumulative_count:0 upper_bound:1216.7361030974075 > bucket:<cumulative_count:0 upper_bound:1460.083323716889 > bucket:<cumulative_count:0 upper_bound:1752.0999884602668 > bucket:<cumulative_count:0 upper_bound:2102.51998615232 > bucket:<cumulative_count:0 upper_bound:2523.023983382784 > bucket:<cumulative_count:0 upper_bound:3027.6287800593404 > bucket:<cumulative_count:0 upper_bound:3633.154536071209 > bucket:<cumulative_count:0 upper_bound:4359.785443285451 > bucket:<cumulative_count:1 upper_bound:5231.74253194254 > bucket:<cumulative_count:52 upper_bound:6278.091038331048 > bucket:<cumulative_count:293 upper_bound:7533.709245997257 > bucket:<cumulative_count:1067 upper_bound:9040.45109519671 > bucket:<cumulative_count:2198 upper_bound:10848.54131423605 > bucket:<cumulative_count:2904 upper_bound:13018.249577083261 > bucket:<cumulative_count:3394 upper_bound:15621.899492499913 > bucket:<cumulative_count:3777 upper_bound:18746.279390999895 > bucket:<cumulative_count:4029 upper_bound:22495.535269199874 > bucket:<cumulative_count:4097 upper_bound:26994.642323039843 > bucket:<cumulative_count:4133 upper_bound:32393.57078764781 > bucket:<cumulative_count:4154 upper_bound:38872.28494517738 > bucket:<cumulative_count:4161 upper_bound:46646.741934212856 > bucket:<cumulative_count:4164 upper_bound:55976.090321055424 > bucket:<cumulative_count:4166 upper_bound:67171.3083852665 > bucket:<cumulative_count:4167 upper_bound:80605.5700623198 > bucket:<cumulative_count:4168 upper_bound:96726.68407478376 > bucket:<cumulative_count:4169 upper_bound:116072.02088974051 > bucket:<cumulative_count:4170 upper_bound:139286.4250676886 > bucket:<cumulative_count:4170 upper_bound:167143.71008122628 > bucket:<cumulative_count:4170 upper_bound:200572.45209747157 > bucket:<cumulative_count:4170 upper_bound:240686.9425169659 > bucket:<cumulative_count:4170 upper_bound:288824.33102035907 > bucket:<cumulative_count:4171 upper_bound:346589.19722443086 > bucket:<cumulative_count:4171 upper_bound:415907.036669317 > bucket:<cumulative_count:4172 upper_bound:499088.4440031804 > bucket:<cumulative_count:4174 upper_bound:598906.1328038165 > bucket:<cumulative_count:4175 upper_bound:718687.3593645798 > bucket:<cumulative_count:4175 upper_bound:862424.8312374958 > bucket:<cumulative_count:4176 upper_bound:1.0349097974849949e+06 > bucket:<cumulative_count:4179 upper_bound:1.2418917569819938e+06 > bucket:<cumulative_count:4183 upper_bound:1.4902701083783926e+06 > bucket:<cumulative_count:4190 upper_bound:1.7883241300540708e+06 > bucket:<cumulative_count:4191 upper_bound:2.1459889560648855e+06 > bucket:<cumulative_count:4201 upper_bound:2.575186747277862e+06 > bucket:<cumulative_count:4206 upper_bound:3.090224096733434e+06 > bucket:<cumulative_count:4206 upper_bound:3.708268916080121e+06 > bucket:<cumulative_count:4206 upper_bound:4.449922699296146e+06 > bucket:<cumulative_count:4206 upper_bound:5.339907239155375e+06 > bucket:<cumulative_count:4206 upper_bound:6.407888686986448e+06 > bucket:<cumulative_count:4206 upper_bound:7.689466424383738e+06 > bucket:<cumulative_count:4206 upper_bound:9.227359709260486e+06 > bucket:<cumulative_count:4206 upper_bound:1.1072831651112583e+07 > bucket:<cumulative_count:4206 upper_bound:1.32873979813351e+07 > bucket:<cumulative_count:4206 upper_bound:1.594487757760212e+07 > bucket:<cumulative_count:4206 upper_bound:1.9133853093122542e+07 > bucket:<cumulative_count:4206 upper_bound:2.296062371174705e+07 > bucket:<cumulative_count:4206 upper_bound:2.7552748454096463e+07 > bucket:<cumulative_count:4206 upper_bound:3.3063298144915752e+07 > bucket:<cumulative_count:4206 upper_bound:3.96759577738989e+07 > bucket:<cumulative_count:4206 upper_bound:4.7611149328678675e+07 > bucket:<cumulative_count:4206 upper_bound:5.713337919441441e+07 > bucket:<cumulative_count:4206 upper_bound:6.856005503329729e+07 > bucket:<cumulative_count:4206 upper_bound:8.227206603995673e+07 > bucket:<cumulative_count:4206 upper_bound:9.87264792479481e+07 > bucket:<cumulative_count:4206 upper_bound:1.1847177509753771e+08 > bucket:<cumulative_count:4206 upper_bound:1.4216613011704522e+08 > bucket:<cumulative_count:4206 upper_bound:1.705993561404543e+08 > bucket:<cumulative_count:4206 upper_bound:2.0471922736854514e+08 > bucket:<cumulative_count:4206 upper_bound:2.4566307284225416e+08 > bucket:<cumulative_count:4206 upper_bound:2.9479568741070503e+08 > bucket:<cumulative_count:4206 upper_bound:3.537548248928459e+08 > bucket:<cumulative_count:4206 upper_bound:4.245057898714152e+08 > bucket:<cumulative_count:4206 upper_bound:5.0940694784569824e+08 > bucket:<cumulative_count:4206 upper_bound:6.112883374148378e+08 > bucket:<cumulative_count:4206 upper_bound:7.335460048978053e+08 > bucket:<cumulative_count:4206 upper_bound:8.802552058773663e+08 > bucket:<cumulative_count:4206 upper_bound:1.0563062470528396e+09 > bucket:<cumulative_count:4206 upper_bound:1.2675674964634075e+09 > bucket:<cumulative_count:4206 upper_bound:1.5210809957560892e+09 > bucket:<cumulative_count:4206 upper_bound:1.825297194907307e+09 > bucket:<cumulative_count:4206 upper_bound:2.1903566338887677e+09 > bucket:<cumulative_count:4206 upper_bound:inf > > timestamp_ms:1658831700000 } was collected before with the same name and label values

The problem was solved by itself after a while but we would like to get some guidance on could we avoid this to hep again in the future.

This seems to be similar to https://github.com/prometheus-community/stackdriver_exporter/issues/103

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named. Start by reviewing the stackdriver_exporter metric collection path and related issue #103, then determine how duplicate Stackdriver metrics can be reproduced; done should include a confirmed cause and documented guidance or a tested fix that prevents scrape failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.