roachprod/failure-injection: increase failure injection test coverage

By intentionally introducing faults, engineers can observe how the system behaves under adverse conditions, identify weaknesses, and ensure that the system can recover gracefully. This will increase test coverage by exposing edge cases and rare conditions that might not occur during normal operation, leading to more robust and resilient systems.

Current failure injection testing is limited and ad hoc when it exists. A failure injection framework should be created to expand testing coverage of these scenarios. See [this design doc](https://docs.google.com/document/d/1iwewT4V_HxXN9Xa-bwngDmwxVqZP3aL1IFvIonjhFWg) for implementation details.

This issue tracks the overall work done, for details on more incremental deliverables, see the linked issues:

- [ ] [FI Library Initial Support](https://github.com/cockroachdb/cockroach/issues/138970)
- [x] #140634
- [ ] FI Planner
- [ ] [Monitor Refactor](https://github.com/cockroachdb/cockroach/issues/118214)
- [ ] FI Controller
- [ ] Report FI events to Monitor
- [ ] Initial Support for Roachtests
- [ ] Teach roachtest monitor to ignore failures caused by FI
- [ ] Support Multiple Clusters
- [ ] Support for DRT
- [ ] Support for mixed version roachtests
- [ ] FI Library more complex failures
- [ ] Performance Assertions

(note: links will be added above as issues are created)

Jira issue: CRDB-46439