oxidecomputer / oxidecomputer/omicron

Metrics: mean_within is not the same as rate, need rate alignment method.

Open
#8,833 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Metrics oximeter oxql
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I have been trying to understand why the data returned by oxql for certain metrics doesn't appear to be correct. Results for sled_data_link:bytes_received appear to be wrong, but instance_network_interface:bytes_received is accurate(edit: also not accurate). If you report on sled_data_link:bytes_received, the shape of the data is correct but the scale is off by a factor of around 6x.

It appears that metrics that originate in propolis are correct, but metrics from sled agent show this problem.

Looking at the underlying data in clickhouse shows one potential reason for this: various data points are being recorded twice. Both in the field metadata, and in the counter data.

SELECT *
FROM fields_string
WHERE timeseries_key = 11128529732522174938

Query id: 1a887728-6bc7-423c-908b-a256a8fa016f

┌─timeseries_name───────────────┬───────timeseries_key─┬─field_name──┬─field_value─┐
│ sled_data_link:bytes_received │ 11128529732522174938 │ kind        │ physical    │
│ sled_data_link:bytes_received │ 11128529732522174938 │ link_name   │ cxgbe1      │
│ sled_data_link:bytes_received │ 11128529732522174938 │ sled_model  │ 913-0000019 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ sled_serial │ BRM42220054 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ zone_name   │ global      │
└───────────────────────────────┴──────────────────────┴─────────────┴─────────────┘
┌─timeseries_name───────────────┬───────timeseries_key─┬─field_name──┬─field_value─┐
│ sled_data_link:bytes_received │ 11128529732522174938 │ kind        │ physical    │
│ sled_data_link:bytes_received │ 11128529732522174938 │ link_name   │ cxgbe1      │
│ sled_data_link:bytes_received │ 11128529732522174938 │ sled_model  │ 913-0000019 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ sled_serial │ BRM42220054 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ zone_name   │ global      │
└───────────────────────────────┴──────────────────────┴─────────────┴─────────────┘

SELECT *
FROM measurements_cumulativeu64
WHERE timeseries_key = 11128529732522174938
ORDER BY timestamp DESC
LIMIT 10

Query id: f01f5770-41d2-4af0-83ac-52707e3aa439

┌─timeseries_name───────────────┬───────timeseries_key─┬────────────────────start_time─┬─────────────────────timestamp─┬──────────datum─┐
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:39:16.105464886 │ 68988141997778 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:39:16.102813033 │ 68988141997679 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:39:06.100306866 │ 68987968764316 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:39:06.097660623 │ 68987968761061 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:38:56.093634082 │ 68987803345334 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:38:56.091277837 │ 68987803340200 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:38:46.090901646 │ 68987630476073 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:38:46.087598811 │ 68987630475974 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:38:36.086132753 │ 68987457048124 │
│ sled_data_link:bytes_received │ 11128529732522174938 │ 2025-06-11 16:42:50.404426319 │ 2025-08-12 22:38:36.082716334 │ 68987457048033 │
└───────────────────────────────┴──────────────────────┴───────────────────────────────┴───────────────────────────────┴────────────────┘

You can see that each datapoint is recorded twice about 3ms apart. The value in datum is correct.

I haven't figured out how to prove this definitively yet but I think that this double data causes oxql to have issues when you aggregate using align mean_within(60s).

The raw data shows the correct rates if queried manually:

SELECT
    toStartOfInterval(timestamp, toIntervalMinute(1)) AS m,
    round((max(datum) - min(datum)) / 60) AS rate
FROM measurements_cumulativeu64
WHERE (timeseries_key = 11128529732522174938) AND ((timestamp >= '2025-08-12 19:00:00') AND (timestamp <= '2025-08-12 19:30:00'))
GROUP BY m
ORDER BY m ASC

Query id: 70f6c920-3bce-438c-aa46-c018ab788273

┌───────────────────m─┬───────rate─┐
│ 2025-08-12 19:00:00 │   14064691 │
│ 2025-08-12 19:01:00 │   14182009 │
│ 2025-08-12 19:02:00 │   14182594 │
│ 2025-08-12 19:03:00 │   66643012 │
│ 2025-08-12 19:04:00 │  159651781 │
│ 2025-08-12 19:05:00 │   14429375 │
│ 2025-08-12 19:06:00 │   14092017 │
│ 2025-08-12 19:07:00 │  172089766 │
│ 2025-08-12 19:08:00 │   13947520 │
│ 2025-08-12 19:09:00 │   14120234 │
│ 2025-08-12 19:10:00 │   15544563 │
│ 2025-08-12 19:11:00 │   14106368 │
│ 2025-08-12 19:12:00 │ 1620416797 │
│ 2025-08-12 19:13:00 │ 1888247403 │
│ 2025-08-12 19:14:00 │ 1903662120 │
│ 2025-08-12 19:15:00 │ 1946602856 │
│ 2025-08-12 19:16:00 │ 1941840609 │
│ 2025-08-12 19:17:00 │ 1944148687 │
│ 2025-08-12 19:18:00 │ 1862297031 │
│ 2025-08-12 19:19:00 │ 1854985828 │
│ 2025-08-12 19:20:00 │ 1852933950 │
│ 2025-08-12 19:21:00 │ 1591121739 │
│ 2025-08-12 19:22:00 │ 1484137878 │
│ 2025-08-12 19:23:00 │   13893543 │
│ 2025-08-12 19:24:00 │   14137592 │
│ 2025-08-12 19:25:00 │   16349895 │
│ 2025-08-12 19:26:00 │   14014514 │
│ 2025-08-12 19:27:00 │   14077108 │
│ 2025-08-12 19:28:00 │   14472012 │
│ 2025-08-12 19:29:00 │   14093051 │
└─────────────────────┴────────────┘

1944148687 is 14.5 gbits/sec and the expected value.

Running the same query through oxql:

get sled_data_link:bytes_received
        | filter timestamp > @2025-08-12T19:00:00 && timestamp < @2025-08-12T19:30:00
        && link_name ~= "cxgbe1" && sled_serial == "BRM42220054"
        | align mean_within(1m)
        | group_by [sled_serial, link_name], sum

returns

link_name (string): cxgbe1
 sled_serial (string): BRM42220054
  2025-08-12T19:01:00Z: 82732559.79621463
  2025-08-12T19:02:00Z: 84688871.67149305
  2025-08-12T19:03:00Z: 84897398.32549168
  2025-08-12T19:04:00Z: 347231064.5319199
  2025-08-12T19:05:00Z: 812300418.3359209
  2025-08-12T19:06:00Z: 85543561.84065337
  2025-08-12T19:07:00Z: 84336928.3815822
  2025-08-12T19:08:00Z: 874666647.7785858
  2025-08-12T19:09:00Z: 83830115.39129297
  2025-08-12T19:10:00Z: 84536133.35660133
  2025-08-12T19:11:00Z: 91568305.03837724
  2025-08-12T19:12:00Z: 84495572.6063845
  2025-08-12T19:13:00Z: 9134030333.216215
  2025-08-12T19:14:00Z: 11118791726.795008
  2025-08-12T19:15:00Z: 11054543030.405613
  2025-08-12T19:16:00Z: 11108611292.24912
  2025-08-12T19:17:00Z: 11118070250.804626
  2025-08-12T19:18:00Z: 11174854814.26545
  2025-08-12T19:19:00Z: 10755800215.056986
  2025-08-12T19:20:00Z: 11061874955.979227
  2025-08-12T19:21:00Z: 11122362761.050976
  2025-08-12T19:22:00Z: 13385112264.910364
  2025-08-12T19:23:00Z: 7646369283.945313
  2025-08-12T19:24:00Z: 83357605.64088883
  2025-08-12T19:25:00Z: 84728996.81025772
  2025-08-12T19:26:00Z: 94095084.20529236
  2025-08-12T19:27:00Z: 83360866.31536402
  2025-08-12T19:28:00Z: 84585494.62588601
  2025-08-12T19:29:00Z: 86141574.8630108
  2025-08-12T19:30:00Z: 75347949.38440287

Where 11118070250 is 82.8 gbps.

During the middle of the test when the values are the largest, the data from oxql is the following times higher than the expected values:

5.840738023
5.678889762
5.720660718
5.718734542
6.000575971
5.798319347
5.969924053

The double data comes out to 12 data points per minute instead of 6, so this could explain why the resulting values are off by a factor of 6.. but I haven't looked into how oxql evaluates mean_within yet.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the oxql query using align mean_within(1m) and compare its results with the provided ClickHouse queries against fields_string and measurements_cumulativeu64. Investigate how duplicate datapoints affect rate alignment, then verify that the aligned output matches the expected raw-data rates without the observed multiplier.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.