commaai / commaai/openpilot

Chestnut-class and RL device-class models cause locationd errors on long drives

Open
#38,505 4 comments 0 reactions 1 assignee Claimed by @adeebshihadeh View on GitHub
bug
Dominant language
Python
Stars
63.7k
Forks
11.4k
Avg merge
17h 53m
Merged PRs (30d)
171

Description

This issue can be seen in [this segment](https://connect.comma.ai/7bc0de9da607c543/000000e5--eeff68d5dc/1401/1414) from @Karnbir.

Not 100% confident but seems like:
- camerad's msgq endpoint for each camera stream has 15 reader slots. These aren't reclaimed when a `VisionIpcClient` is destroyed
- When the car passes through 5m/s, the UI switches the displayed camera by creating a new `VisionIpcClient`, which consumes another reader slot for the target stream
- Before this error, there had been 13 fcam to ecam transitions during the drive. Those + modeld reader + encoderd reader filled all 15 ecam slots
- When the car slowed below 5m/s again, the UI registered a new client which reset all all existing reader slots, including modeld's ecam reader
- With the RL model, modeld finished processing frame 28087 ~4ms after the reset. The smaller, 7ms faster model, probably would have let modeld recieve the ecam frame before the reset
- When modeld tried to read the next ecam frame it had to re-register it's reader. This skipped over frame 28088 so modeld ran on ecam frame 28089 and fcam frame 28088
- On the next iteration, modeld ran on frame 28090 from both streams. It detected the dropped fcam frame 28089 and marked `cameraOdometry` with `valid=false`
- locationd tolerates one bad camera-odometry input as long as it reject it itself. If it comes in marked as `valid=false`, locationd immediately makes `inputsOK=false`

Seems like a lot went wrong here. This might cause issues on big models as well but not sure how specific the timing issue is.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.