lowRISC / lowRISC/opentitan

[ci,fpga] Identify and fix the cause of flaky FPGA manuf tests

Open
#30,946 0 comments 0 reactions 2 assignees View on GitHub

@nsw-lowrisc is already working on this.

Since Aug 6, 2026.

Dominant language
SystemVerilog
Stars
3.6k
Forks
1.1k
Avg merge
2d 22h
Merged PRs (30d)
141

Description

PR https://github.com/lowRISC/opentitan/pull/30912 disables a set of unstable and flaky manufacturing tests entirely to increase the stability of the CI. Currently the tests are completely disabled.

While disabling the tests in CI brings short-term relieve, the underlying issue has to be identified and fixed.

Various smaller efforts have been made towards the identification or the solution of this problem; #29555 captures one of them. So far we see that the flakiness is directly tied to the use and mode switch of the JTAG port. We could further identify that if such an issue arises, a simple reset (using the FPGA's button reset) is not enough to recover functionality leading to the possibility of some non-resettable state that preserves the issue between runs. A bitstream reload (and the following global reset) clears the issue.

We started (automated) systematic data collection of the CI runs. These efforts are collected in this sheet.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.