Incremental runs

Nobody has claimed this yet.

At the moment newer DVSim runs replace previous runs in a destructive way. Directories from a previous run are renamed (with a timestamp generated at replacement time) and new results or in their place.

Instead of doing that, we could make the system additive. For each block IP level test there is a target number of seeds to run. This can be overridden via cli arguments, with either a fixed seed, or a more limited set of random seeds.

In the fixed seed case, there is no need to rerun an individual seed if it's already been run for that test. Likewise when generating a report, it would be nice if the new report would report based on all the seed results it has... not just the ones that ran in the last run.

This would mean it would be possible to do things like, run a full regression suite. Identify a test that we want to run some more seeds on. Run those extra seeds and regenerate the reports including those seeds.

Both the pass/fail rate and the coverage should be able to be incrementally added.

This behaviour may not always be what you want. Sometimes you might want to do an A/B test and compare the results. So maybe store a run ID as part of the job result metadata, then give the option of having a column per run (for easier comparison) or merge the results in a cumulative way? Run ID could be a simple time stamp or optionally a label provided as an argument on the CLI... giving more meaningful column names.

The issue names no files, tests, or entry points. Start by locating the code that replaces prior run directories, tracks seed results, and generates reports; done means supporting additive seed runs and cumulative pass/fail and coverage reporting while preserving an option to compare separate runs.

Written by the indexing model from the issue text.