[otbn, dv] Modelling SW errors during delayed escalation

**TL;DR**
The timing optimization proposed in #28345 and implemented in #28815 delays the handling of certain fatal errors by one cycle. The escalation / start of the secure wipe is therefore delayed by one cycle for some hardware fatal errors but not for software error based escalations. This creates challenges in how to model this delay for escalation tests because whether there is a delay to the secure wipe depends on the randomly generated test program. This issue describes the problem in more detail and explains solutions. It also describes why solution 3 was implemented for now.

**Problem details**
The timing optimization proposed in #28345 and implemented in #28815 delays the handling of certain fatal errors by one cycle. The escalation / start of the secure wipe is therefore delayed by one cycle for some hardware fatal errors but not for software error based escalations. The triggered secure wipe due to a HW error differs in two points compared to a secure wipe triggered by a SW error.
Firstly, the [`INSN_CNT`](https://opentitan.org/book/hw/ip/otbn/doc/registers.html#insn_cnt) register is reset during the first few cycles and secondly, the OTBN goes into the locked state at the end. In case a hardware escalation happens during an ongoing software secure wipe the current secure wipe continues but `INSN_CNT` is reset and the OTBN locks up at the end.

The delayed escalation results in challenges for DV tests which cover the various fatal error sources. These tests currently have to tell the OTBN simulator that an escalation is starting. This is implemented by calling `cfg.model_agent_cfg.vif.send_err_escalation(err_val)` in the cycle the escalation happens (`err_val` can indicate a SW or HW error). See also [here](https://github.com/lowRISC/opentitan/blob/acfc7179ee5b0080cfc31b718d4f67c0a8b9922e/hw/ip/otbn/dv/uvm/env/seq_lib/otbn_ctrl_redun_vseq.sv#L314) for an example.

Whereas delaying the hardware triggered escalations by one cycle is easy to model (simply call `send_err_escalation()` one cycle later), it is tricky to model the special case where an injection not only leads to a hardware based error but also to a SW error. This is for example the case if the test case injects an error into the register file during a branch instruction and therefore an invalid branch target is computed. This will result in a `bad_insn_addr_err` SW error. In such a case, OTBN first escalates due to a SW error and in the following cycle due to the delayed HW error. The first escalation will start a secure wipe and the second one will change it to a "HW error secure wipe" where `INSN_CNT` is reset and OTBN locks up.

On a first glance a simple solution would be to call `send_err_escalation()` twice with one cycle delay. However, the problem is that the injection happens during the execution of a random instruction as for each test run a new random binary is generated. As of this an injection will not always result in a SW error and thus the DV code must handle this case dynamically based upon the current program and internal state.

The problem is that the current program is not easily available inside the DV code / simulator (to see which instruction we are executing when injecting the error) and it requires detailed knowledge of internal OTBN logic to check whether the current injection will lead to a SW error. Duplicating such OTBN details (error checks) in the DV code is against the goal of having "high-level" DV code.

**Solutions**
Possible solutions to handle the SW errors during the "waiting cycle" are:

- Solution 1: Extend the OTBN simulator to model also all possible injection errors in detail such that the model can resolve whether a SW error arises or not.
- The DV code would not call `send_err_escalation()` but trigger a new command where the injection information is passed. The model itself would then escalate as required.
- Solution 2: Allow the RTL model and simulator to diverge after injecting an error until the secure wipe has finished.
- In this case, the test would only call `send_err_escalation()` once for the expected HW escalation. Any checks where the RTL is compared to the simulator model would be ignored from the cycle an error is injected until the secure wipe has finished.
- If the injection leads to an SW error the RTL and the simulator will go out-of-sync during the secure wipe (the model would finish the secure wipe one cycle too late). Some logic is required to synchronize them when both have finished their secure wipe.
- Solution 3: The DV code probes the two error signals directly from the RTL (`software_err` and `fatal_software_err` in the `otbn_controller.sv`) which indicate whether a SW error is present or not. Based upon this information the DV code will call `send_err_escalation()` accordingly. The HW escalation is always sent. Probing these signals allows to keep the SW error logic to the RTL but still keep the model in sync with the RTL.

**Discussion**
The solution 1 would be the cleanest but requires to fully model the OTBN as the error logic depends on actual implementation details (i.e., it would required to model all the branch target computations or memory address computations to then also implement the actual error checks like misaligned address checks etc.). This would make the OTBN model even more complex (it is already a cycle accurate model but with some simplifications) and the efforts would be relatively huge as the current error handling is based upon the [error bits](https://opentitan.org/book/hw/ip/otbn/doc/registers.html#err_bits) only. Therefore, this solution is not optimal.

The 2nd solution could probably be implemented with less effort than solution 1 but disabling the RTL-simulator comparison could suppress errors of the secure wipe. I haven't fully explored all consequences but it seems to be an acceptable trade-off but with quite some effort.

The 3rd solution is the one implemented in #28815 and it avoids the high overhead of replicating complex error logic within the DV model. Sampling the SW error signals directly from the RTL model to influence the behaviour of the DV model seems to contradict the principle of separating the models. However, it does not compromise any tests because the probed signals are used solely to synchronize the models and do not affect whether a HW based escalation is triggered (which is the actual behavior a test wants to check). It simply allows to send an additional escalation signal to the model in case the RTL detects a SW error. Due to the dependency on the RTL, this SW error handling approach may not be used to check the correctness of the SW error implementation but tests which inject errors which should result in a HW escalation usually do not check the SW error handling. As of this it is not a problem if the SW error handling is wrongly implemented in the RTL. In case the RTL misses an error the DV model will not escalate but the test still checks whether the HW escalation happened. In case the RTL raises an error despite there should not be one, the RTL and simulator will escalate too early but it is still checked whether there was an HW escalation or not.

Due to the simplicity of solution 3 and as it does what it should it was implemented for now. However, we should come back to this topic and discuss it with DV folks whether there is a smarter solution.

Start with the escalation handling in hw/ip/otbn/dv/uvm/env/seq_lib/otbn_ctrl_redun_vseq.sv and the software_err and fatal_software_err signals in otbn_controller.sv. Review the three synchronization approaches described here with the DV team, then define and test an agreed approach that keeps RTL and simulator behavior synchronized during delayed escalations.

Written by the indexing model from the issue text.