commaai / commaai/openpilot

test_models: add a test that fuzzes the tx messages

Open
#32,425 10 comments 0 reactions 0 assignees View on GitHub
bounty enhancement
Dominant language
Python
Stars
63.7k
Forks
11.4k
Avg merge
17h 53m
Merged PRs (30d)
171

Description

Currently we compare the car states from panda and openpilot to ensure that both parse CAN messages with the same 1) validity checks and 2) correctness/no bugs. That let us fix quite a few mismatches in behavior between the two: https://github.com/commaai/openpilot/pull/30443

Now we want to fuzz the tx messages as well, to discover bugs related to controlsAllowed in panda, and general tx safety. Here is one such mismatch that we should aim to catch: https://github.com/commaai/panda/pull/1948

See the first PR linked above for rx to know where to start. Bounty is for a merged PR that adds fuzz testing for sent openpilot messages that
1) detects mismatches in tx (sent messages) logic between openpilot and panda,
1) extensible (generic, and easy to expand with more state),
2) as performant as our current fuzzy testing, and
3) covers as many cases as possible (catches a few bugs along the way, if there are any).

Contributor guide

Open the contributing guide

Research direction

Start with the first linked PR and its RX fuzzing implementation to locate the existing test entry points and understand how performance is measured. Extend that approach to fuzz sent openpilot messages against panda, covering controlsAllowed and general TX safety. Done means the test detects TX mismatches, is generic to extend with more state, remains performant, and catches relevant bugs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
embedded-iot, security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.