[ci] HyperDebug firmware update flakiness
- Dominant language
- SystemVerilog
- Stars
- 3.6k
- Forks
- 1.1k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 141
Description
### Description
It was discovered that a source of FPGA CI flakiness was from HyperDebug firmware update. Example: https://dev.azure.com/lowrisc/opentitan/_build/results?buildId=159507&view=logs&j=6d7ef521-d3a1-575b-de1b-a7342dcf1a8e&t=5fbf7d75-9dee-5f42-7084-95893df8eb52
The failure pattern is:
* HyperDebug firmware is updated
* USB disconnects (by inspecting kernel log)
* Job failed with "no device" found
* USB reconnects (by inspecting kernel log)
The HyperDebug USB device takes ~3 seconds to be rediscovered, which is enough for jobs to fail.
The HyperDebug firmware update code checks if the correct version is already running, but we have diverging HyperDebug version in different branches, causing frequent update to happen.
I think the following actions should be taken
- [ ] Ensure HyperDebug firmware versions are identical across branches - address the symptom by reducing HyperDebug updates needed.
- [x] ~~Maybe by default only upgrade but not downgrade HyperDebug versions? I am not sure there's compatibility guarantees @jesultra~~ Reverted as this prevents downgrade of problematic versions
- [x] In OT-lib, after firmware update, confirm the HyperDebug is alive. It appears that we have some provision (1 second wait + USB enumeration?) but it does not cover all code paths and is likely not sufficient.
Contributor guide
Research direction
Start by tracing the OT-lib HyperDebug firmware-update paths and the FPGA CI jobs that invoke them. Compare the HyperDebug firmware versions used across branches, then verify the post-update device-aliveness handling covers each update path and prevents jobs from running before USB rediscovery; reproduce the failure using the linked CI log.
Written by the indexing model from the issue text.
Assessment
- Domain
- ci-cd, embedded-iot
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100