Python Comparison Scripts

NVBench has a work-in-progress JSON output format and I'm working on a very basic python script to compare two JSON files.

We should grow this functionality into a more complete set of analysis tools. At minimum, this should cover the features provided by Google Benchmarks' excellent comparison scripts.

If anyone is interested in writing some python to help with this, let me know. I'll update this issue once I have finalized the JSON output format.

Basic Regression Testing

• P0: Compare two json files:
  compare.py baseline.json test.json
• P0: Specify a custom error threshold:
  compare.py --gpu-threshold 5 baseline.json test.json
  (gpu-threshold, cpu-threshold, batch-threshold)
• P2: Run a benchmark executable and compare with a json
  compare.py baseline.json --run test.exe -b 3 -a T=[I32,U64] -a Elements[pow2]=30

These should:

• Compare the benchmarks with the same name + config.
• Print abs/rel changes for cpu/gpu/batch measurements.
• Highlight any entries that exceed a threshold time.
• Return an error code if any exceed thresholds.

Analysis modes

Compare benchmarks with different names. Answers questions:

• How much faster is benchmark X for input type T vs. U for a variety of input sizes?
• Does Algorithm X take more time to run than Algorithms Y for the same inputs?

These will need some way of specifying the sets of configurations to compare. Google benchmark has worked out a general syntax for specifying this, we should adapt what they've done to use the NVBench axis syntax.

Output

Ideally markdown formatted, similar to NVBench's default output.

References

• Google Benchmark compare.py docs
• NVBench's axis specification syntax

Start with the proposed compare.py commands and the Google Benchmark comparison-script documentation, then review NVBench's axis specification syntax and the current JSON output format. The format is still described as work in progress, so confirm its final structure before implementing. Done should include regression comparisons, threshold handling, analysis modes, and markdown output as specified.