MetOffice / MetOffice/Vernier

[Custom Issue]: Scaling scores - comparing two sets of Vernier output.

Open
#247 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C++
Stars
6
Forks
11
Avg merge
18h 7m
Merged PRs (30d)
1

Description

### Contact Details

_No response_

### Are there any linked Issues or Pull Requests?

Similar in nature to #246, but ranking regions according to a different parameter.

### Brief description

It'd be useful to support a command-line tool which takes two sets of Vernier output and orders profiled regions according to how well they are scaling. The two runs would use different resources, such as 1 versus 2 MPI ranks in the sample below.

e.g. `vernier-scaling /path/to/runA /path/to/runB`.

```
Baseline file: baseline-file.out, procs: 0001, threads: 01
Test file: test-file.out, procs: 0002, threads: 01

Rank Routine name Time difference (s)
------- ------------------------------------------------------- -------------------
1 anti_scaling_routine -0.500
2 no_scaling_routine 0.000
3 sub_linear_scaling_routine 0.500
4 perfect_scaling_routine 1.000
5 super_linear_scaling_routine 1.500
```

The "scaling score" is calculated as:

```math
S =
\left( \frac{p_1 t_1 T_1}{p_2 t_2 T_2} - 1 \right) /
\left( \frac{p_1 t_1}{p_2 t_2} - 1 \right)
+ 1
```
where $p$ is the number of MPI tasks, $t$ is the number of threads per MPI task and $T$ is the elapsed time. The index 1 refers to the reference
(baseline) time and resources; index 2 is the test time and resources.

| Score value | Meaning |
|--|--|
| $S < 0$ | Region is anti-scaling. |
| $S = 0$ | Region shows no scaling. |
| $0 < S < 1$ | Region shows sub-linear scaling. |
| $S = 1$ | Region shows ideal scaling. |
| $S > 1$ | Region shows super-linear scaling. |

There would need to be a **cut-off** mechanism in terms of absolute time taken: a region that takes $2 \mu s$, up from $1 \mu s$ would be anti-scaling but might clutter the output with information that isn't useful.

### Further details of the issue.

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names no files, tests, or CLI entry point; start by reviewing the implementation context of similar issue #246 and how Vernier output records resources and elapsed times. Done means a command-line comparison of two runs that ranks regions by the specified scaling score and applies an agreed absolute-time cutoff.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.