oxidecomputer / oxidecomputer/hubris

Duplicate hardware metrics on sensor ambiguity

Open
#2,634 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

Consider the follow oxql query:

> oxide --profile r3 experimental system timeseries query --query 'get hardware_component:temperature | filter component_id == "PSU0" && timestamp > @now() - 1m | last 3'

{
  "tables": [
    {
      "name": "hardware_component:temperature",
      "timeseries": [
        {
          "fields": {
            "chassis_model": {
              "type": "string",
              "value": "913-0000003"
            },
            "component_kind": {
              "type": "string",
              "value": "mwocp68"
            },
            "chassis_serial": {
              "type": "string",
              "value": "BRM45220018"
            },
            "chassis_revision": {
              "type": "u32",
              "value": 8
            },
            "description": {
              "type": "string",
              "value": "PSU 0 MCU"
            },
            "hubris_archive_id": {
              "type": "string",
              "value": "a7cec194aa73a90f"
            },
            "chassis_kind": {
              "type": "string",
              "value": "power"
            },
            "rack_id": {
              "type": "uuid",
              "value": "48759651-d754-43e2-ad49-64dde3adc1c2"
            },
            "sensor": {
              "type": "string",
              "value": "psu0mcu"
            },
            "gateway_id": {
              "type": "uuid",
              "value": "dfeef10e-0dda-432d-a06a-f103456881ce"
            },
            "component_id": {
              "type": "string",
              "value": "PSU0"
            },
            "slot": {
              "type": "u32",
              "value": 0
            }
          },
          "points": {
            "timestamps": [
              "2026-08-11T16:23:50.684568706Z",
              "2026-08-11T16:23:50.684583584Z",
              "2026-08-11T16:23:50.684595417Z"
            ],
            "values": [
              {
                "metric_type": "gauge",
                "values": {
                  "type": "double",
                  "values": [
                    15.75,
                    45.25,
                    40.75
                  ]
                }
              }
            ]
          }
        },
        {
          "fields": {
            "hubris_archive_id": {
              "type": "string",
              "value": "a7cec194aa73a90f"
            },
            "chassis_revision": {
              "type": "u32",
              "value": 8
            },
            "chassis_serial": {
              "type": "string",
              "value": "BRM45220018"
            },
            "sensor": {
              "type": "string",
              "value": "psu0mcu"
            },
            "component_kind": {
              "type": "string",
              "value": "mwocp68"
            },
            "component_id": {
              "type": "string",
              "value": "PSU0"
            },
            "rack_id": {
              "type": "uuid",
              "value": "48759651-d754-43e2-ad49-64dde3adc1c2"
            },
            "chassis_model": {
              "type": "string",
              "value": "913-0000003"
            },
            "description": {
              "type": "string",
              "value": "PSU 0 MCU"
            },
            "chassis_kind": {
              "type": "string",
              "value": "power"
            },
            "gateway_id": {
              "type": "uuid",
              "value": "7cff00d5-b96b-4f9f-b6dd-cb8f8d5e42ba"
            },
            "slot": {
              "type": "u32",
              "value": 0
            }
          },
          "points": {
            "timestamps": [
              "2026-08-11T16:23:49.683031412Z",
              "2026-08-11T16:23:49.683044597Z",
              "2026-08-11T16:23:49.683053594Z"
            ],
            "values": [
              {
                "metric_type": "gauge",
                "values": {
                  "type": "double",
                  "values": [
                    15.875,
                    45.25,
                    40.75
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  ]
}

In particular, the timestamps of the last three values:

            "timestamps": [
              "2026-08-11T16:23:50.684568706Z",
              "2026-08-11T16:23:50.684583584Z",
              "2026-08-11T16:23:50.684595417Z"
            ],

These timestamps are the same down to the millisecond. But we sample the sensor every 1s. Why are we getting multiple almost-identical timestamps for a single sensor, with very different values?

This seems to be happening because the psu0mcu sensor on the mwocp68 actually includes multiple sensors, and we don't distinguish among them in hubris. If the different sensors measured the same thing, this might not matter much, but I believe they're actually measuring different things (ambient vs hot-spot temperatures). So I think these metrics are basically uninterpretable until we include the full sensor names.

It looks like we would fix this by updating metadata around https://github.com/oxidecomputer/hubris/blob/9e553c76105497896a725b89558b46fda0ee10c6/app/psc/base.toml. It would also be helpful to include some kind of check to avoid duplicate sensor names if possible, because we also have a similar conflict in hardware_component:fan_speed, also for the mwocp68.

More selfishly, I noticed this because I tried to ingest these metrics into prometheus, which only has millisecond timestamp precision, and hit errors due to conflicting timestamps. That isn't necessarily an Oxide bug, but I think the missing sensor information actually is a bug.

Possibly relevant to your interests @hawkw @evan-oxide.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting app/psc/base.toml and the metadata for the mwocp68 psu0mcu sensor, then reproduce the hardware_component:temperature query from the issue. Trace how sensor names become metric series and check the similar hardware_component:fan_speed conflict; done means the distinct sensors have full, non-duplicated names and the reported metrics are distinguishable.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.