llnl / llnl/smith

Regression testing for examples

Open
#1,394 0 comments 0 reactions 0 assignees View on GitHub
CS testing
Dominant language
C++
Stars
245
Forks
36
Avg merge
8d 23h
Merged PRs (30d)
4

Description

We should create some mechanism to track changes of example results over time.

https://en.wikipedia.org/wiki/Regression_testing

It may be possible to track result values through adiak, and upload the examples similar to how we do with benchmarks.

```cpp
SERAC_SET_METADATA("result", result);
```

Using SPOT or Hatchet, we can then graph change in this value over time.

We should also see how other projects do regression testing or something similar.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reviewing how examples produce results and how benchmarks are uploaded, then investigate whether adiak can track result values and how SPOT or Hatchet could graph them. Done means there is an agreed mechanism for tracking example-result changes over time, informed by comparable projects.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.