oxidecomputer / oxidecomputer/omicron

Oximeter: some crucible_upstairs queries broken on r2

Open
#10,189 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Here's an innocuous oxql query that doesn't work on r2, as of this writing:

$ joshcarp@j0sh ~/c/oxidereceiver (jmcarp/remote-write) [1]> oxide --profile r2 experimental system timeseries query --query "get crucible_upstairs:write | filter timestamp > @now() - 15m"
error
Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "1dab0d96-4696-4462-973b-3ef4a4852f7c", "content-length": "466", "date": "Mon, 30 Mar 2026 19:21:01 GMT"}; value: Error { error_code: Some("InvalidRequest"), message: "Cannot compute a delta, the timestamp of the next sample has a new start time, or overlaps with the last processed sample. \n epoch start time = 2026-03-17 20:18:41.611418768 UTC\n last timestamp = [2026-03-30 19:20:45.226962784 UTC, 2026-03-30 19:20:55.227868499 UTC]\n new timestamp = [2026-03-17 21:23:37.295694225 UTC, 2026-03-30 19:06:05.191748974 UTC]", request_id: "1dab0d96-4696-4462-973b-3ef4a4852f7c" }

The identical query works on r3, and interestingly also works on r2 if we append | last 1 to the end.

I don't think the relevant code has changed recently, so I'm guessing we're dealing with some unexpected data that's specific to r2, but haven't had time to look yet.

cc @bnaecker

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the experimental system timeseries query command on r2 and r3 with the crucible_upstairs:write query, then compare it with the r2 query ending in | last 1. Start from the reported InvalidRequest timestamp ranges and identify the r2 data or processing difference; done means the original query works on r2 without the workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.