E3SM-Project / E3SM-Project/zstash

[Feature]: Improve efficiency of `zstash check`

Open
#446 4 comments 0 reactions 1 assignee Claimed by @forsyth2 View on GitHub
semver: new feature
Dominant language
Python
Stars
11
Forks
10
Avg merge
14h 18m
Merged PRs (30d)
1

Description

### How will this affect the next version number?

New feature (increment MINOR version)

### Is your feature request related to a problem?

Yes, @chengzhuzhang noted [here](https://github.com/E3SM-Project/zstash/pull/427#issuecomment-4615985477):
> There have been complaints about zstash check being slow.

Please note that this issue is similar, but not identical to #410. That issue was closed because the `--tars` option was a sufficient solution to the problem it considered:
> We need zstash check to be able to immediately jump to a specified point

That is, that issue was resolved by running on fewer tars. This issue asks if we can get more tars-per-time-unit.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

Once #427 is merged, profiling should be added for `zstash check` as well. A few important considerations for that:
1. The existing performance records don't have `check` data, so we will need to allow for backwards compatibility in the performance scripts.
2. #427 already produces 5 relatively busy plots just for `create`, `update`, and `extract` (both sequential and parallel), so we will need to think about the best way to visualize `check`. Include it on the same output? A separate figure? Does it make sense to compare it to how long the full `extract` took?
3. Make sure that running `zstash check` doesn't interfere with other results. Considering we're running `extract twice (sequential and parallel), I think this should be feasible.

That is, this issue should only be considered resolved once we have both:
1. A more efficient `zstash check`
2. Performance profiling scripts that confirm this is so.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.