Layr-Labs / Layr-Labs/d-inference
[v0.7.10][fan helper] M4 Max becomes permanently unsupported after implausible GPU reading and failed Auto restore
- Dominant language
- Go
- Stars
- 522
- Forks
- 113
- Avg merge
- 17h 26m
- Merged PRs (30d)
- 111
Description
## Component
Provider fan-control helper (`provider-swift`, privileged helper / SMC recovery)
## What happened?
Darkbloom's experimental fan helper worked on an M4 Max MacBook Pro, then one GPU sensor returned `-4.0 C`. The helper correctly rejected the implausible reading, but failed to restore either fan to automatic mode, failed to clear the global `Ftst` ownership flag, exited, failed its first launchd recovery, and then restarted in a persistent `unsupported` state.
Hours later, unprivileged diagnostics still detected all eight plausible GPU sensors and both fans, while the privileged helper reported no sensors, no fans, and `mode: unsupported`. The provider lease remained active. The helper did not retry hardware discovery or recover ownership.
## Environment
- Darkbloom: `0.7.10`
- macOS: `26.5.2` (`25F84`)
- Hardware: MacBook Pro `Mac16,5`, Apple M4 Max, 128 GB unified memory
- Workload: `gemma-4-26b-qat-4bit`, public coordinator serving
- Capability: `darkbloom-fan-helper-v1`
- Helper protocol: `1`
- Fan control was explicitly enabled and initially working
- Policy: 80% target, engage at 70 C, release at 65 C
Documentation identifies this M4 Max family as read-only discovery validated, with active control not yet exercised:
- https://github.com/Layr-Labs/d-inference/blob/v0.7.10/docs/provider/fan-control.md
- https://github.com/Layr-Labs/d-inference/pull/536
- https://github.com/Layr-Labs/d-inference/releases/tag/v0.7.10
## Exact failure sequence (EDT, July 14)
```text
17:00:57.032 fan policy tick failed: GPU sensor Tg1k reported implausible temperature -4.0 C
17:00:57.036 automatic fan restore failed: fan operation failed (explicit automatic restore); rollback also failed: restoreMode fan 0: fan mode remained 1; restoreMode fan 1: fan mode remained 1; clearFtst global: Ftst remained 1
17:01:14.596 automatic fan restore failed: fan operation failed (explicit automatic restore); rollback also failed: restoreMode fan 0: fan mode remained 1; restoreMode fan 1: fan mode remained 1; clearFtst global: Ftst remained 1
17:01:14.668 fan helper startup failed: fan ownership recovery failed: restoreMode fan 0: fan mode remained 1; restoreMode fan 1: fan mode remained 1; clearFtst global: Ftst remained 1
17:01:24.794 fan helper started
```
Later launchd state:
```text
state = running
runs = 7
last exit code = 1
```
## Persistent split state
`darkbloom fan status --json` then had this shape:
```json
{
"installed": true,
"loaded": true,
"diagnostic": {
"chip": "M4",
"supported": true,
"gpuTemperatures": ["8 plausible readings, hottest about 78 C"],
"fans": ["2 fans in auto mode"]
},
"helper": {
"chip": "M4",
"enabled": true,
"providerActive": true,
"mode": "unsupported",
"gpuSensorKeys": [],
"fans": []
}
}
```
The helper process was alive and accepted authenticated status requests, so this was not an SSH, launchd, provider-lease, signature, or helper-connectivity failure. Fresh diagnostic discovery succeeded while the helper's cached hardware inventory remained empty.
## Thermal impact
After the helper degraded, the hottest sampled GPU sensor reached **102.7 C** at 19:41:17 EDT while both fans remained under macOS automatic control. Four samples in the two-hour evidence window were at or above 88 C. The provider did not crash in that window, but unattended serving lost the explicitly enabled cooling policy.
## Expected behavior
The documented fail-safe should return fans to macOS Auto when a sensor becomes invalid. After the transient reading becomes plausible again, the helper should either:
1. rediscover hardware and resume its provider lease/policy automatically, or
2. expose a durable actionable error and a documented non-destructive recovery command.
One transient invalid sensor should not leave an otherwise supported machine permanently `unsupported` while seven other GPU sensors and both fans remain readable.
## Actual behavior
- Auto restore failed while `F0Md`, `F1Md`, and `Ftst` remained in manual/owned state.
- The first crash-recovery launch failed.
- A later launch stayed running but initialized with no hardware inventory.
- The helper never retried discovery after readings recovered.
- `providerActive` remained true, but no manual cooling policy could engage.
## Steps to reproduce
The transient `-4.0 C` value has not yet been forced deterministically.
1. On `Mac16,5`, enable the v0.7.10 helper from the provider account.
2. Configure 80% at 70 C and run sustained Gemma inference.
3. Poll `darkbloom fan status --json` while the helper owns both fans.
4. When `Tg1k` returns a transient implausible value, inspect the `io.darkbloom.fan` unified log and launchd restart state.
5. After readings recover, compare `diagnostic` and `helper` inventories.
## Verified recovery
Running the normal signed configure path once recovered the helper:
```text
sudo darkbloom fan configure --speed 80 --temperature 70
```
Darkbloom kickstarted the privileged helper. Launchd advanced from run 7 to run 8, the PID changed, and the previous exit code changed from 1 to 0. The helper repopulated all eight GPU keys and both fans, reacquired the active provider lease, entered `manual`, and applied the 80% target. No provider restart, uninstall, model change, or reboot was needed.
This confirms that the hardware remained discoverable and controllable. The defect is the degraded helper's failure to retry process-lifetime hardware discovery after the transient sensor/recovery event.
## Questions / suggested fixes
1. Can hardware discovery retry after a fail-closed sensor event?
2. Can one invalid key be quarantined while other validated GPU keys remain available, with a conservative fallback target?
3. Why did explicit Auto restore and startup journal recovery both fail to change `F0Md`, `F1Md`, and `Ftst`?
4. Can `fan status` expose `lastError` and distinguish unsupported hardware from supported hardware with failed initialization?
5. Please add an M4 Max test covering engage, invalid-sensor fail-safe, restart, rediscovery, and lease reacquisition.
## Evidence and privacy
The full fan status snapshots, launchd state, thermal samples, and unified logs are preserved and can be shared privately.
This report omits the machine serial, username, hardware UUID, provider/account identifiers, hostnames, IP addresses, and tailnet details.
## Confirmation
- [x] I searched existing issues and did not find a report containing this `Tg1k = -4.0 C`, failed Auto restore, empty cached helper inventory, and successful signed-configure recovery sequence.
Contributor guide
Research direction
Start with the provider-swift privileged helper's recovery and hardware-discovery entry points, using `darkbloom fan status --json` and the `io.darkbloom.fan` unified log to compare diagnostic and helper inventories. Review the launchd restart path and `darkbloom fan configure --speed 80 --temperature 70` recovery behavior. Done should include the proposed M4 Max test covering invalid-sensor fail-safe, restart, rediscovery, and lease reacquisition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- desktop-dev, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100