oxidecomputer / oxidecomputer/omicron
Timeseries schema mismatch for `virtual_machine:incomplete_check`
Open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
While investigating #6923, I looked at the oximeter logs, and see that there's a schema mismatch for the virtual_machine:incomplete_check timeseries:
actual = TimeseriesSchema { timeseries_name: TimeseriesName("virtual_machine:incomplete_check"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "failure_reason", field_type: String, source: Metric, description: "" }, FieldSchema { name: "instance_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "nexus_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "project_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "silo_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_agent_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_agent_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "sled_agent_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "vmm_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2024-10-23T21:45:38.254218130Z }
collector_id = da510a57-3af1-4d2b-b2ed-2e8849f27d8b
collector_ip = fd00:1122:3344:10a::3
expected = TimeseriesSchema { timeseries_name: TimeseriesName("virtual_machine:incomplete_check"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "instance_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "nexus_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "project_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "rack_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "reason", field_type: String, source: Metric, description: "" }, FieldSchema { name: "silo_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_agent_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "sled_agent_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "sled_agent_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "vmm_id", field_type: Uuid, source: Target, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2024-06-20T17:45:08.409040686Z }
file = oximeter/db/src/client/mod.rs:956
id = 9a4a2991-4e16-4b2b-82e8-266bb31c2e20
sample = Sample { measurement: Measurement { timestamp: 2024-10-23T21:45:38.190023779Z, datum: CumulativeU64(Cumulative { start_time: 2024-10-23T20:00:48.375346243Z, value: 1 }) }, timeseries_name: TimeseriesName("virtual_machine:incomplete_check"), timeseries_version: 1, target: FieldSet { name: "virtual_machine", fields: {"instance_id": Field { name: "instance_id", value: Uuid(db4aca25-6828-4d7e-8744-83336110e82b) }, "nexus_id": Field { name: "nexus_id", value: Uuid(2898657e-4141-4c05-851b-147bffc6bbbd) }, "project_id": Field { name: "project_id", value: Uuid(829b10e4-19a0-4498-8d4a-ebafa097d307) }, "rack_id": Field { name: "rack_id", value: Uuid(de608e01-b8e4-4d93-b972-a7dbed36dd22) }, "silo_id": Field { name: "silo_id", value: Uuid(7bd7623a-68ed-4636-8ecb-b59e3b068787) }, "sled_agent_id": Field { name: "sled_agent_id", value: Uuid(87c2c4fc-b0c7-4fef-a305-78f0ed265bbc) }, "sled_agent_ip": Field { name: "sled_agent_ip", value: IpAddr(fd00:1122:3344:108::1) }, "sled_agent_port": Field { name: "sled_agent_port", value: U16(12345) }, "vmm_id": Field { name: "vmm_id", value: Uuid(bf8faa13-5c4a-4e21-86a8-e2fa6d198c3a) }} }, metric: FieldSet { name: "incomplete_check", fields: {"failure_reason": Field { name: "failure_reason", value: String("unreachable") }} } }
I don't have any further data, and am going to tackle the higher-priority #6923 first for obvious reasons.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at oximeter/db/src/client/mod.rs:956 and inspect how the actual and expected schemas for virtual_machine:incomplete_check are produced and compared. Trace the schema definitions and producer named by the issue, then verify that the timeseries no longer reports a field-name mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100