Feature Request: System Simulator for Fault Managment

It would be extremely useful to have a deterministic testing harness for the sitrep planning logic (aka, the diagnosis engine).

In our production system, we'll have the following phases:

• The database will store system state
• The running system will continually be updating database state based on real observations (this includes, but is not limited, to inventory)
• Sitrep preparation will load a variety of inputs from the database
• Sitrep planning will take these inputs, as well as a parent sitrep, and create a new sitrep as an output
• Sitrep execution will take the sitrep output, and propagate the intent of that sitrep out to the rest of the system (typically through writing back to the database, but it would also be valid to e.g. make an API call to an arbitrary service.

For testing planning, this introduces a lot of dependencies for adequately running the system under test, attempting to inject faults, etc.

By building a simulator, we should basically have a system which can stub out the preparation and execution phases with arbitrary rust structures. In essence:

• Create a struct which keeps track of "what does the system look like right now, as far as the FM system is concerned"? For example, this will include an "inventory collection" created synthetically.
• We should be able to transform this "system view" into "the set of all inputs to the planning phase".
• Once we have created a new sitrep, we should be able to use that sitrep to create updates back to the system. (one example: sitreps will declare that ereports should be marked seen. The set of ereports that we have will be simulated. so "marking an ereport seen" involves reading a sitrep, and updating our simulated system state, so the next iteration of "preparation" sees new state).

Comparison with the Reconfigurator

nexus/reconfigurator/simulation is the equivalent for the reconfigurator. The most directly comparable thing to the system I've described above is SimSystem (it contains a SystemDescription, which is just a bunch of plain data to emulate the layout of a rack, as well as a map of all known inventory collections).

The reconfigurator has some bells and whistles on their simulated system that are pretty handy:

• The SimSystem is wrapped in a SimState, which acts like a node...
• ... RNG is managed via SimRng to have determinism...
• ... and a bunch of SimStates are managed by the Simulator struct, in src/main.rs.

Each of these simulated states are an immutable object, which lets us track the relationship between them, identify diffs between them, see how they've changed, etc. Since each simulated state is modeled as immutable, it allows us to arbitrarily undo/redo operations too, which is kinda neat.

The reconfigurator-cli is a REPL that lets you interact with a Simulator - you can load example systems, run the planner to see how it would modify a system, and interactively modify system state. This is basically just an interactive version around what is used in the tests (see: nexus/reconfigurator/planning/tests/). However, we do also use this REPL for a lot of "EXPECTORATE"-style testing - dev-tools/reconfigurator-cli/tests/input is a directory full of REPL commands, where we do both normal operations ("set a target release, see what happens") and contorted things ("try to add a sled which has no disks, see what happens"). We then often dump out "what blueprints get generated in response to these system states", and can easily see if subsequent PRs change these outputs.