commaai / commaai/openpilot

comma 4: recurring ADSP_EFAILED + CAM-SENSOR shutdown cluster causes cascading camerad/manager restarts (distinct from #38270)

Open
#38,518 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
63.7k
Forks
11.4k
Avg merge
17h 53m
Merged PRs (30d)
171

Description

# Title
comma 4: recurring ADSP_EFAILED + CAM-SENSOR shutdown cluster causes cascading camerad/manager restarts (distinct from #38270)

## Summary

Over several days of normal driving on a comma 4 (mici hardware), I'm seeing a recurring kernel-level fault cluster: a burst of Qualcomm audio-DSP failures immediately followed by camera-sensor shutdown errors, occasionally severe enough to cascade into multiple full `manager.py` restarts within minutes. Confirmed via direct `dmesg` analysis over a multi-day window, not a one-off. This looks related to the general "camera malfunction on comma 4" cluster (e.g. #38270), but the specific kernel signature is different, so filing separately in case it's a distinct root cause worth tracking on its own.

## Signature

```
__q6asm_cmd: DSP returned error[ADSP_EFAILED] opcode 68557
q6asm_callback: cmd = ... returned error = ...
CAM_ERR: CAM-SENSOR: cam_sensor_shutdown: 629 failed destroying dhdl
q6asm_send_cal: cal_block is NULL
```

The audio-DSP errors and camera-sensor shutdown errors consistently appear together, in tight bursts (bursts of dozens of lines within a few seconds), not as isolated one-off lines.

## Scale (from direct dmesg analysis over the current boot, ~3.3 days of continuous uptime)

- 862 occurrences of `ADSP_EFAILED` opcode 68557
- 123 occurrences of `cam_sensor_shutdown ... failed destroying dhdl`
- 82 occurrences of `q6asm_send_cal: cal_block is NULL`
- Grouped into 71 distinct burst clusters across 4 calendar days: 18, 8, 29, and 16 bursts/day respectively — day-to-day variable, not a steady background rate.

On the two worst days, burst clusters were tight enough (repeated bursts within a few minutes of each other) to cascade into multiple full `manager.py` process-stack restarts — e.g. 5 restarts in ~82 minutes on one drive, with the gaps between restarts shrinking over the course of the drive (39min → 15min → 17min → 11min).

## Downstream user-visible symptoms (inferred from the restart cascade, not independently confirmed as separate bugs)

Each restart briefly re-triggers normal startup-transition alerts that would otherwise only appear once per drive: a brief silent "TAKE CONTROL" flash (`manualRestart`/low-priority, not a real disengage), transient `locationd`/torque-parameter-not-ready states, and occasional spurious chime re-announcements — all consistent with genuine repeated process-stack restarts, not separate underlying bugs.

## What's been ruled out

- **Thermal**: checked `deviceState.thermalStatus` directly across two separate severe cascade events (different days) — stayed `ok` in every sample, zero exceptions. Actual temperatures were *falling*, not rising, over the course of both cascades (e.g. one cascade ran 62-91°C early, dropping to 50-59°C later in the same cascade while instability continued). Ruled out as the driver, at least for this unit.
- **Memory leak**: `deviceState.memoryUsagePercent` shows a normal fill-to-plateau pattern on every restart, resetting cleanly each time — no progressive growth.
- **Software regression on our end**: confirmed via `git reflog` that zero commits were deployed to this device across the entire multi-day window this kept recurring in — the fault rate varied day to day with the exact same code running throughout.
- **Not specific to any fork/customization**: found (via a separate historical check of archived route telemetry) a comparably severe restart-cascade cluster on **2026-07-18**, confirmed via that route's own `initData.gitCommit` to be running the literal, unmodified `release-mici` branch tip — before any of our own commits existed. Also directly diffed our current `release-mici`-based HEAD against sunnypilot's live `release-mici` branch tip today: byte-identical, zero drift.

## Relation to #38270

Likely the same general hardware-generation issue cluster (comma 4 camera/audio kernel driver flakiness), but the specific signature differs: #38270's attached log shows `CAM-ISP` congestion/`buf_done` failures, while this report is `CAM-SENSOR` shutdown + `ADSP` audio failures occurring together. Filing separately since I can't confirm they share a root cause, but flagging the relation in case it's useful for whoever's looking at the broader cluster.

## Environment

- Device: comma 4 (mici hardware)
- Branch: fork of `sunnypilot`, `release-mici` base (confirmed byte-identical to sunnypilot's current `release-mici` branch tip, zero customization to the relevant subsystems), plus unrelated application-layer changes on top (lateral/longitudinal advisory features, no changes to `system/camerad/` or audio-related code)
- AGNOS: 18.4
- This looks hardware/firmware-level rather than car-specific, so car platform omitted as not load-bearing here

Contributor guide

Open the contributing guide

Research direction

Start with the supplied dmesg signatures and restart timeline, then inspect manager.py and the referenced system/camerad/ area. Compare the ADSP_EFAILED and CAM-SENSOR shutdown bursts with the linked #38270 report and AGNOS 18.4 context. Done means establishing whether this is a distinct reproducible fault and documenting a confirmed root cause or actionable fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
embedded-iot, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.