oxidecomputer / oxidecomputer/omicron
Metrics: hardware_component:poll_error_count query returns timeseries not found error in some cases
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
A query against the component polling error metrics returns the following HTTP 400 error on rack2:
$ oxide experimental timeseries query --query 'get hardware_component:poll_error_count'
error
Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "9692e4d9-0633-470b-b90b-6652d8e75ece", "content-length": "170", "date": "Fri, 27 Sep 2024 14:29:56 GMT"}; value: Error { error_code: Some("InvalidRequest"), message: "Timeseries not found for: hardware_component:poll_error_count", request_id: "9692e4d9-0633-470b-b90b-6652d8e75ece" }
The problem was also seen with a rackettle but wasn't with rack3 (where the query returned one row of data). It would appear that when there isn't any polling error, instead of inserting a zero count, the producer skips it altogether.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with oxide experimental timeseries query --query 'get hardware_component:poll_error_count' on a rack with no polling errors, then trace the producer that emits this metric. Confirm the behavior on racks with and without errors. Done means the query returns a zero count instead of a timeseries-not-found error when no polling errors occurred.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100