Proto Fleet does not correctly handle firmware updates to/from 3rd-party firmwares
- Dominant language
- Go
- Stars
- 55
- Forks
- 16
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 87
Description
## Summary
Proto Fleet does not correctly support reflashing miners between stock and 3rd-party firmwares (Braiins OS+, VNish, LuxOS, Marathon, ePIC). After a reflash, the device record stored by fleet drifts from reality and only recovers if the operator deletes and re-pairs the miner. There is also no first-class "update firmware" operation — the capability is reserved but unimplemented.
A second, related gap: a reflash can change which **plugin** is responsible for the device (e.g., stock Bitmain → Braiins moves the device from the `antminer` plugin to the `asicrs` plugin), and Proto Fleet has no mechanism for that handoff.
## Use case
A fleet operator reflashes a paired miner between firmwares — for example:
- Bitmain S21 stock → Braiins OS+ (to gain native Stratum V2)
- Bitmain S21 with Braiins → stock (rolling back)
- WhatsMiner stock → VNish, etc.
Expected: Proto Fleet detects the new firmware on next telemetry/reconnect and updates the device's plugin assignment, manufacturer, firmware version, default credentials, and firmware-keyed capabilities to match.
Actual: the device record drifts. The operator must delete and re-pair to recover correct state, and even that only works if the new firmware happens to be discoverable by a different plugin.
## Where the drift comes from (within-plugin, asicrs)
The asicrs plugin considers `manufacturer` to mean "firmware manufacturer" — set once at `PairDevice` time from `firmware_manufacturer(detect_variant(...))` ([plugin/asicrs/src/driver.rs:546-551](https://github.com/block/proto-fleet/blob/main/plugin/asicrs/src/driver.rs#L546-L551)).
After pairing, nothing refreshes it:
1. The reconnect/probe path refreshes `model` and control/config `caps` from live `MinerData` ([plugin/asicrs/src/device.rs:113-127](https://github.com/block/proto-fleet/blob/main/plugin/asicrs/src/device.rs#L113-L127), [device.rs:208-213](https://github.com/block/proto-fleet/blob/main/plugin/asicrs/src/device.rs#L208-L213)) but never touches `self.info.manufacturer`.
2. Firmware-derived caps are never re-probed. `probe_capabilities()` ([capabilities.rs:122-145](https://github.com/block/proto-fleet/blob/main/plugin/asicrs/src/capabilities.rs#L122-L145)) only reads control/config caps via the `Miner` trait — it does not consult `device_info.firmware` to re-derive variant.
3. `GetCapabilitiesForModel` (PR #129) keys `native_stratum_v2` purely off `req.manufacturer` ([driver.rs:649-651](https://github.com/block/proto-fleet/blob/main/plugin/asicrs/src/driver.rs#L649-L651)), which the server pulls from its stored device record.
4. `probed_caps_for` filters representative devices by stored `device.info.manufacturer` ([driver.rs:189](https://github.com/block/proto-fleet/blob/main/plugin/asicrs/src/driver.rs#L189)).
Default credentials are also looked up by manufacturer/family at pair time, so a reflashed miner may end up with the wrong default creds in stored state.
## Where the drift gets worse (cross-plugin reflashes)
A firmware reflash can change which plugin is responsible for managing the device. The plugins partition the firmware space:
- **antminer plugin** handles stock Bitmain firmware only, and explicitly rejects non-stock at discovery — see `isNonStockFirmware` and [plugin/antminer/internal/driver/driver.go:258](https://github.com/block/proto-fleet/blob/main/plugin/antminer/internal/driver/driver.go#L258): `non-stock firmware detected (...), skipping antminer plugin`.
- **asicrs plugin** handles Braiins / VNish / LuxOS / Marathon / ePIC on Bitmain hardware (and other ASIC families).
A device's plugin assignment is set at pair time and pinned in `discovered_device.driver_name` ([server/internal/domain/pairing/service.go:1155](https://github.com/block/proto-fleet/blob/main/server/internal/domain/pairing/service.go#L1155) treats `driver_name` mismatch as a reconciliation barrier). Nothing in fleet ever changes that assignment. So when a Bitmain S21 paired under the `antminer` plugin is reflashed to Braiins:
- The `antminer` plugin can no longer talk to it (non-stock firmware is explicitly rejected). Telemetry, control, and pool config all fail.
- The `asicrs` plugin does not know the device exists — its stored `driver_name` is still `antminer`.
- Re-discovery via the `asicrs` plugin would find the device on the network, but the existing record blocks reuse because the driver names don't match.
- The only way to recover is delete-and-re-pair, which loses any per-device history (groups, schedules, naming, telemetry continuity).
The reverse (Braiins → stock) has the same problem in the opposite direction.
## Where the firmware-update operation is missing
`CAP_FIRMWARE` is hardcoded to `false` in `probe_capabilities` ([capabilities.rs:141-142](https://github.com/block/proto-fleet/blob/main/plugin/asicrs/src/capabilities.rs#L141-L142)) with the comment:
> Do not advertise firmware updates until the update_firmware RPC is implemented.
So fleet does not support initiating a reflash from the UI today — operators must reflash out-of-band (miner UI or direct flashing tool), which is what produces the drift described above.
## Impact
- Stale manufacturer / firmware version displayed in the UI.
- Wrong default credentials cached for the device after reflash.
- Firmware-keyed capabilities go stale. The first concrete instance is `native_stratum_v2` (introduced in #129, dormant until the SV2 PR3 lands). When PR3 ships, capability-dependent routing/UI can treat a stock miner as native-SV2 capable or fail to recognize a newly-Braiins miner, causing failed pool reconfiguration and avoidable downtime.
- Cross-plugin reflashes silently break the device — no telemetry, no control — until the operator deletes and re-pairs, which loses history.
- No supported workflow for reflashing from within Proto Fleet at all.
## Recommended scope
Four increments, smallest first:
1. **Refresh manufacturer + firmware version on reconnect (within-plugin).** In `ensure_connected`, recompute `firmware_manufacturer(detect_variant(make, firmware))` from live `MinerData.device_info` and update `self.info.manufacturer` and the persisted device record alongside `model`/`caps`. Closes the data-drift gap for out-of-band reflashes that stay within the same plugin.
2. **Re-derive firmware-keyed caps from live data.** Move the SV2 derivation (and any future firmware-keyed caps) into `probe_capabilities`, computing variant from `device_info.firmware`. Drop the manufacturer-string check in `GetCapabilitiesForModel`. Removes the dependency on stale request metadata.
3. **Cross-plugin reflash handoff.** Define a server-level reconciliation path for "this device's firmware now belongs to a different plugin." Likely shape: on connect failure (or on a periodic probe via the existing IP scanner), if another plugin can discover and identity-verify the same device by serial/MAC, transfer the device record's `driver_name` to the new plugin and migrate cached secrets/credentials. Identity verification by serial/MAC keeps it safe against IP reassignment.
4. **First-class firmware update RPC.** Implement the `update_firmware` RPC the comment in `capabilities.rs:141` is reserving for, so reflashes can be initiated and tracked through Proto Fleet rather than only happening out-of-band. This also lets the system anticipate the cross-plugin handoff in (3) instead of detecting it after the fact.
(1) and (2) together fix within-plugin silent drift. (3) fixes the cross-plugin failure mode. (4) is the larger product gap.
Contributor guide
Research direction
Start with plugin/asicrs/src/device.rs, capabilities.rs, and driver.rs to trace reconnect state, firmware-derived capabilities, and stored manufacturer data; then read server/internal/domain/pairing/service.go for driver assignment reconciliation. Done means within-plugin reflashes refresh persisted firmware state and capabilities, while cross-plugin reflashes preserve the device record through a verified handoff; the firmware update RPC is a larger follow-on scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- api, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100