Errors with the same canonical code collapse into one row, hiding distinct vendor errors
- Dominant language
- Go
- Stars
- 55
- Forks
- 16
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 84
Description
## Summary
A Proto OS miner reporting two distinct errors shows only one error in fleet. The error dedup key cannot distinguish two different vendor error codes that map to the same canonical `MinerError`, so the second error overwrites the first instead of being stored as a separate row.
## Observed behavior
Miner `172.16.25.198` reports two errors via the Proto OS errors API:
```json
[
{"source": "rig", "slot": null, "error_code": "03:0010", "timestamp": 1781107676, "message": "Hashboard in bay 1 is incompatible with current configuration"},
{"source": "rig", "slot": null, "error_code": "03:0020", "timestamp": 1781107544, "message": "Bay 2 has insufficient cooling"}
]
```
Fleet shows only one:
```json
{
"canonicalError": "MINER_ERROR_VENDOR_ERROR_UNMAPPED",
"summary": "Bay 2 has insufficient cooling",
"causeSummary": "Unhandled error code: rig/03:0020",
"severity": "SEVERITY_INFO",
"firstSeenAt": "2026-06-10T16:07:56Z"
}
```
Note the row is a chimera of both errors: `firstSeenAt` (16:07:56Z) is 03:0010's timestamp, while `summary`/`causeSummary` are 03:0020's.
## Root cause
1. Neither `03:0010` nor `03:0020` is in `rigErrorMappings` (`plugin/proto/internal/device/errors.go`), so both map to `MINER_ERROR_VENDOR_ERROR_UNMAPPED` (severity INFO). The raw vendor code only survives inside the `causeSummary` text.
2. Both errors have `slot: null`, so neither gets a `ComponentID`/`ComponentType`.
3. `UpsertError` (`server/internal/domain/stores/sqlstores/error.go`) dedups on `(org_id, device_id, miner_error, component_id, component_type)` via `GetOpenErrorByDedupKey` (`server/sqlc/queries/errors.sql`). That key is identical for both errors, so the second runs `UpdateOpenError` instead of inserting, overwriting `summary`, `cause_summary`, and `last_seen_at` on the first. `first_seen_at` is the one field the update does not touch, hence the chimera.
## Impact
- Any miner reporting 2+ unmapped vendor errors (with no slot) only ever shows one in fleet; which one wins depends on iteration order of each poll.
- The same collision exists for mapped errors: two distinct vendor codes mapping to the same canonical error on the same component collapse into one row.
## Proposed fix
Make the dedup key vendor-code aware. The `errors` table already has a `vendor_code` column, but the plugin SDK `DeviceError` (`server/sdk/v1/errors/types.go`) has no vendor-code field. The proto plugin should carry the raw code (e.g. `rig/03:0020`) into a vendor code field/attribute, and `GetOpenErrorByDedupKey` should include it in the dedup key, at minimum for `VENDOR_ERROR_UNMAPPED` where the canonical code carries no identity.
Alternative (narrower): add `03:0010` / `03:0020` to `rigErrorMappings` so they get distinct canonical errors. Fixes these two codes but not the general class.
## Related observation
`rigErrorMappings` is keyed entirely by symbolic names (`"LowHashRate"`, `"PsuPowerNoGood"`), but the real device sends numeric `NN:NNNN` codes. If current Proto OS firmware only emits numeric codes, none of the existing rig mappings ever match in production and every rig error lands in the unmapped bucket. The fake-proto-rig test double does not simulate errors, so this would not surface in tests. Worth verifying against real firmware.
Contributor guide
Research direction
Start with UpsertError in server/internal/domain/stores/sqlstores/error.go and GetOpenErrorByDedupKey in server/sqlc/queries/errors.sql, then trace DeviceError in server/sdk/v1/errors/types.go and the Proto OS handling in plugin/proto/internal/device/errors.go. Reproduce the two unmapped errors and inspect how vendor_code is stored and used. Done means distinct vendor errors no longer overwrite one another, including mapped errors sharing a canonical code, with coverage for the collision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100