block / block/proto-fleet

Errors with the same canonical code collapse into one row, hiding distinct vendor errors

Open
#424 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
55
Forks
16
Avg merge
1d 17h
Merged PRs (30d)
84

Description

## Summary

A Proto OS miner reporting two distinct errors shows only one error in fleet. The error dedup key cannot distinguish two different vendor error codes that map to the same canonical `MinerError`, so the second error overwrites the first instead of being stored as a separate row.

## Observed behavior

Miner `172.16.25.198` reports two errors via the Proto OS errors API:

```json
[
{"source": "rig", "slot": null, "error_code": "03:0010", "timestamp": 1781107676, "message": "Hashboard in bay 1 is incompatible with current configuration"},
{"source": "rig", "slot": null, "error_code": "03:0020", "timestamp": 1781107544, "message": "Bay 2 has insufficient cooling"}
]
```

Fleet shows only one:

```json
{
"canonicalError": "MINER_ERROR_VENDOR_ERROR_UNMAPPED",
"summary": "Bay 2 has insufficient cooling",
"causeSummary": "Unhandled error code: rig/03:0020",
"severity": "SEVERITY_INFO",
"firstSeenAt": "2026-06-10T16:07:56Z"
}
```

Note the row is a chimera of both errors: `firstSeenAt` (16:07:56Z) is 03:0010's timestamp, while `summary`/`causeSummary` are 03:0020's.

## Root cause

1. Neither `03:0010` nor `03:0020` is in `rigErrorMappings` (`plugin/proto/internal/device/errors.go`), so both map to `MINER_ERROR_VENDOR_ERROR_UNMAPPED` (severity INFO). The raw vendor code only survives inside the `causeSummary` text.
2. Both errors have `slot: null`, so neither gets a `ComponentID`/`ComponentType`.
3. `UpsertError` (`server/internal/domain/stores/sqlstores/error.go`) dedups on `(org_id, device_id, miner_error, component_id, component_type)` via `GetOpenErrorByDedupKey` (`server/sqlc/queries/errors.sql`). That key is identical for both errors, so the second runs `UpdateOpenError` instead of inserting, overwriting `summary`, `cause_summary`, and `last_seen_at` on the first. `first_seen_at` is the one field the update does not touch, hence the chimera.

## Impact

- Any miner reporting 2+ unmapped vendor errors (with no slot) only ever shows one in fleet; which one wins depends on iteration order of each poll.
- The same collision exists for mapped errors: two distinct vendor codes mapping to the same canonical error on the same component collapse into one row.

## Proposed fix

Make the dedup key vendor-code aware. The `errors` table already has a `vendor_code` column, but the plugin SDK `DeviceError` (`server/sdk/v1/errors/types.go`) has no vendor-code field. The proto plugin should carry the raw code (e.g. `rig/03:0020`) into a vendor code field/attribute, and `GetOpenErrorByDedupKey` should include it in the dedup key, at minimum for `VENDOR_ERROR_UNMAPPED` where the canonical code carries no identity.

Alternative (narrower): add `03:0010` / `03:0020` to `rigErrorMappings` so they get distinct canonical errors. Fixes these two codes but not the general class.

## Related observation

`rigErrorMappings` is keyed entirely by symbolic names (`"LowHashRate"`, `"PsuPowerNoGood"`), but the real device sends numeric `NN:NNNN` codes. If current Proto OS firmware only emits numeric codes, none of the existing rig mappings ever match in production and every rig error lands in the unmapped bucket. The fake-proto-rig test double does not simulate errors, so this would not surface in tests. Worth verifying against real firmware.

Contributor guide

Open the contributing guide

Research direction

Start with UpsertError in server/internal/domain/stores/sqlstores/error.go and GetOpenErrorByDedupKey in server/sqlc/queries/errors.sql, then trace DeviceError in server/sdk/v1/errors/types.go and the Proto OS handling in plugin/proto/internal/device/errors.go. Reproduce the two unmapped errors and inspect how vendor_code is stored and used. Done means distinct vendor errors no longer overwrite one another, including mapped errors sharing a canonical code, with coverage for the collision.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.