lowRISC / lowRISC/opentitan

[verilator, tests] Timed-out Verilator test targets in CI

Open
#24,184 2 comments 0 reactions 1 assignee Claimed by @AlexJones0 View on GitHub
Dominant language
SystemVerilog
Stars
3.6k
Forks
1.1k
Avg merge
2d 22h
Merged PRs (30d)
141

Description

### Description

All Verilator test targets have been run in public CI in preparation for the addition of a weekly/nightly Verilator job. Of the current 390 targets, 67 tests timed out when run using `--test_timeout=2400,2400,3600,-1` (compared to the standard `60,300,900,3600`). All tests that timed out hence took at least 40 minutes, and so should probably not be run in a CI environment on Verilator. See related issue #24182 for Verilator tests that failed.

A separate PR will mark all of these Verilator test targets as `manual` (to avoid running in CI), but we need to determine for each test whether they genuinely ran out of time (i.e. allocating more time will allow the test to succeed) and as such should stay marked `manual`, or whether they timed out due to an error, and as such should be marked `broken` and/or fixed. The aim here is that any Verilator test target that is not marked as broken should be able to be manually run by overriding the test timeout, even if it might take hours to complete.

To run any of these tests, after setting up Verilator for Opentitan, run

```
./bazelisk.sh test -t- --test_output=streamed
```

The `--test_timeout=` Bazel flag can be used to manually override the timeout for these tests.

The 67 timed-out Verilator test targets that need checking are:

- [x] `sw/device/silicon_creator/lib:otbn_boot_services_functest_sim_verilator`
- [x] `sw/device/silicon_creator/lib/sigverify:mod_exp_ibex_functest_hardcoded_sim_verilator`
- [x] `sw/device/silicon_creator/lib/sigverify:mod_exp_ibex_functest_wycheproof_sim_verilator`
- [x] `sw/device/silicon_creator/lib/sigverify:rsa_verify_functest_sim_verilator`
- [x] `sw/device/silicon_creator/lib/sigverify:sigverify_dynamic_functest_hardcoded_sim_verilator`
- [x] `sw/device/silicon_creator/lib/sigverify:sigverify_dynamic_functest_wycheproof_sim_verilator`
- [x] `sw/device/silicon_creator/rom/e2e/weak_straps:sw_strap_value_sim_verilator`
- [x] `sw/device/tests:alert_handler_lpg_sleep_mode_pings_test_sim_verilator`
- [ ] `sw/device/tests:alert_handler_reverse_ping_in_deep_sleep_test_sim_verilator`
- [ ] `sw/device/tests:aon_timer_sleep_wdog_sleep_pause_test_sim_verilator`
- [x] `sw/device/tests:csrng_edn_concurrency_test_sim_verilator`
- [x] `sw/device/tests:edn_auto_mode_sim_verilator`
- [x] `sw/device/tests:entropy_src_fw_observe_many_contiguous_test_sim_verilator`
- [x] `sw/device/tests:entropy_src_fw_override_test_sim_verilator`
- [ ] `sw/device/tests:example_sival_sim_verilator`
- [x] `sw/device/tests:otp_ctrl_mem_access_test_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_deep_sleep_all_reset_reqs_test_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_deep_sleep_all_wake_ups_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_deep_sleep_por_reset_test_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_normal_sleep_all_reset_reqs_test_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_normal_sleep_all_wake_ups_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_normal_sleep_por_reset_test_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_random_sleep_all_reset_reqs_test_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_random_sleep_all_wake_ups_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_sleep_wake_5_bug_test_sim_verilator`
- [ ] `sw/device/tests:pwrmgr_smoketest_sim_verilator`
- [ ] `sw/device/tests:rv_core_ibex_epmp_test_functest_sim_verilator`
- [ ] `sw/device/tests:rv_core_ibex_isa_test_test_unlocked0_sim_verilator`
- [ ] `sw/device/tests:rv_core_ibex_mem_test_test_unlocked0_sim_verilator`
- [ ] `sw/device/tests:sensor_ctrl_wakeup_test_sim_verilator`
- [ ] `sw/device/tests:sleep_pin_mio_dio_val_test_sim_verilator`
- [ ] `sw/device/tests:sleep_pin_retention_test_sim_verilator`
- [ ] `sw/device/tests:spi_device_flash_smoketest_sim_verilator`
- [ ] `sw/device/tests:spi_device_tpm_tx_rx_test_sim_verilator`
- [ ] `sw/device/tests:spi_passthru_test_sim_verilator`
- [ ] `sw/device/tests:sram_ctrl_sleep_sram_ret_contents_no_scramble_test_sim_verilator`
- [ ] `sw/device/tests:sram_ctrl_sleep_sram_ret_contents_scramble_test_sim_verilator`
- [ ] `sw/device/tests:uart_baud_rate_test_sim_verilator`
- [ ] `sw/device/tests:uart_loopback_test_sim_verilator`
- [ ] `sw/device/tests:uart_parity_break_test_sim_verilator`
- [ ] `sw/device/tests:uart_tx_rx_test_sim_verilator`
- [x] `sw/device/tests/autogen:plic_all_irqs_test_0_sim_verilator`
- [x] `sw/device/tests/autogen:plic_all_irqs_test_20_sim_verilator`
- [x] `sw/device/tests/crypto:aes_functest_sim_verilator`
- [x] `sw/device/tests/crypto:aes_gcm_functest_sim_verilator`
- [x] `sw/device/tests/crypto:ecdh_p256_sideload_functest_sim_verilator`
- [x] `sw/device/tests/crypto:ecdh_p384_functest_sim_verilator`
- [x] `sw/device/tests/crypto:ecdh_p384_sideload_functest_sim_verilator`
- [x] `sw/device/tests/crypto:ecdsa_p256_sideload_functest_sim_verilator`
- [x] `sw/device/tests/crypto:ecdsa_p384_functest_sim_verilator`
- [x] `sw/device/tests/crypto:ecdsa_p384_sideload_functest_sim_verilator`
- [x] `sw/device/tests/crypto:hmac_multistream_functest_sim_verilator`
- [x] `sw/device/tests/crypto:kdf_hmac_ctr_functest_sim_verilator`
- [x] `sw/device/tests/crypto:kdf_kmac_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_2048_encryption_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_2048_key_from_cofactor_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_2048_keygen_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_2048_signature_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_3072_encryption_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_3072_keygen_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_3072_signature_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_3072_verify_functest_wycheproof_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_4096_encryption_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_4096_keygen_functest_sim_verilator`
- [x] `sw/device/tests/crypto:rsa_4096_signature_functest_sim_verilator`
- [ ] `sw/device/tests/pmod:i2c_host_fram_test_sim_verilator`
- [x] `third_party/coremark/top_earlgrey:coremark_test_sim_verilator`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.