General system for outputting custom metrics for BenchExec, etc
@sim642 is already working on this.
Since Dec 9, 2021.
- Dominant language
- OCaml
- Stars
- 252
- Forks
- 90
- Avg merge
- 4d 1m
- Merged PRs (30d)
- 19
Description
BenchExec allows tools to output special values that can easily be shown as additional columns in the tables:
Overwriting the function
get_value_from_outputwill allow you to add<column>tags with custom values to your table-definition files, andtable-generatorwill extract the respective values from the output of your tool using this function.
More information at https://github.com/sosy-lab/benchexec/blob/0e0f79cf8d7cc52bf7e7c9a4c622f8d070c855d7/doc/table-generator.md#column-features and https://github.com/sosy-lab/benchexec/blob/0e0f79cf8d7cc52bf7e7c9a4c622f8d070c855d7/doc/tool-integration.md#writing-a-tool-info-module.
Instead of hardcoding a multitude of special-purpose regexes into the tool-info module, which would have to be submitted each time we want to add more, it would be better to have special output syntax in Goblint (and a module for abstracting it) to print these such that the tool-info module can pick them up unchanged. This should be separate from the user-readable output (such that one can be changed without affecting the other) and probably optional.
A consistent output format for such statistics, etc wouldn't be useful for just BenchExec, but also our own benchmarking scripts that currently grep out a variety of specific outputs, which is error-prone when the user-readable output is changed.
And in BenchExec instead of just displaying them in columns, it would also be easy to then make comparative plots against any metric, not just CPU time.
Metrics
- Solver vars and evals
- Solver aborts (#364)
- Race counts (total, safe, vulnerable?, unsafe)
- Uncalled functions count
- Missing function definitions count
- Line counts (total logical, dead, live)
- Message counts by severity (or even category/tag?)
Feel free to suggest any others.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.