The-OpenROAD-Project / The-OpenROAD-Project/OpenROAD
Convert high-churn .ok/.defok golden tests to intent-based assertions
Nobody has claimed this yet.
- Dominant language
- Verilog
- Stars
- 3.1k
- Forks
- 1k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 136
Description
Description
The ask here is to board up the broken windows and get the rock out of the shoe, not convert everything to pristine C++ unit-tests.
Integration tests that diff full logs or DEFs against golden files (.ok, .defok) record current behavior rather than test intent. Any change that perturbs placement, routing, or log output regenerates these files wholesale, and concurrent PRs that touch the same goldens conflict with each other even when their changes are unrelated.
Data from the last 30 days of master (2026-08-01 to 2026-08-31):
- 142 PR landings; 161 commits touched golden files; 280 distinct golden files changed.
- 10 merge commits resolved both-sides changes on golden files (branches include
grt_cugr_jumper_insertion,grt_save_engine_choice,dpl-enhance-initial-snap,dpl-remove-negotiation-padding,ppl_pdn_boundary_1_slots,bazel-flow-metrics-check). - Two currently open PRs are in conflicting state: #11120, #10661.
Golden tests ranked by (PR landings touching their goldens) x (evidence of both-sides merge resolution) x (golden size):
| # | Test (family) | Goldens | Size | 30-day signal |
|---|---|---|---|---|
| 1 | test/upf_aes |
.ok + .defok | 1.3k + 109k lines | 3 landings; both-sides merge resolution |
| 2 | src/cts/test/array_max_wl (+ array, array_ins_delay, array_repair_clock_nets, array_no_blockages) |
.ok + .defok | 58k-line .defok | 3 landings; in 2 both-sides merges |
| 3 | src/dpl/test/cell_on_block2 (+ mirror1) |
.ok + .defok | 46k-line .defok | 2 landings each |
| 4 | src/grt/test/repair_antennas_*_cugr family (~8 tests) |
.ok + .defok | 3.5k-line .defoks | 2-3 landings each; in 3 both-sides merges |
| 5 | src/gpl/test/medium01-06, macro01-03, large01-02 (11 tests) |
.ok | 110-140 lines each | 2 landings each, regenerated as a block |
| 6 | test/upf_test |
.ok | 227 lines | 2 landings; both-sides merge resolution |
| 7 | src/odb/test/replace_hier_mod1 (+ replace_hier_mod6) |
.ok | 761 lines | 2-3 landings; both-sides merge resolution |
| 8 | src/ram/test/make_8x8_2r1w_sky130 |
.ok + .defok | small | 2 landings; both-sides merge resolution |
| 9 | src/grt/test/incremental_repair_cugr (+ 4 sibling cugr tests) |
.ok | 70-165 lines | 3 landings |
| 10 | src/pad/test/make_bterm_pins_from_bumps, src/ppl/test/obstruction_boundary, src/ppl/test/pdn_*_stripes* |
.ok + .defok | small-mid | 3 commits each |
test/*.metrics_limits files (jpeg/aes/gcd/ibex) have the highest raw churn (4-8 changes each) but are numeric guardbands rather than output diffs; they are listed here for completeness, not as conversion candidates.
Suggested solution
Convert the tests above, in roughly this order, from full-output golden diffs to assertions on the properties the test exists to verify, queried through the OpenDB/STA Tcl APIs. Where the intent of a test genuinely is the log output (message counts, specific warnings), the log check stays.
The win this request is after is eliminating .ok/.defok churn in merges. Pure C++ unit tests — dependencies confined to the module, mocked data, seconds-scale compile+test cycles — would be a further step in the same direction and would earn extra bonus points, but they are not the concern here and not what is being asked for.
#11152 is a worked example for test/upf_aes: it replaces the DEF diff and a REGIONS-block text-parsing workaround with direct dbRegion queries (region existence, type, boundary coordinates in DBUs), and removes upf_aes.defok/upf_aes.regionsok plus the Bazel-specific exceptions they required. Caveats: the Tcl was AI-drafted and did not get a full human review; the PR was closed after twice accumulating merge conflicts on the golden files it removes.
Alternatives considered
- Regenerating goldens on every conflict: current practice; the data above is the result.
- C++ unit tests instead of Tcl flow tests: covers serialization, not flow integration; discussed in #11152.
Reproduction of the survey
# golden files changed per PR landing, last 30 days
git log origin/master --first-parent --since="30 days ago" --name-only --format='%H' \
-- '*.ok' '*.defok' '*.vok' '*.regionsok' '*.rptok' | grep -E '\.(ok|defok|vok|regionsok|rptok)$' | sort | uniq -c | sort -rn
# merges whose result differs from both parents on golden files (resolved both-sides changes)
git log origin/master --merges --since="30 days ago" --format='%H' | \
while read s; do git diff-tree --cc --name-only "$s" | tail -n +2 | \
grep -E '\.(ok|defok|vok|regionsok|rptok)$' | sed "s|^|$s |"; done
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with test/upf_aes and the worked example in #11152; inspect its .ok/.defok files and the OpenDB/STA Tcl APIs used there. Compare the listed test families and identify the properties each currently records. Done means intent-based assertions replace the relevant golden diffs, with obsolete goldens and required Bazel exceptions removed while intentional log checks remain.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, testing-qa
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100