E3SM-Project / E3SM-Project/zstash
[Feature]: Improve efficiency of `zstash check`
- Dominant language
- Python
- Stars
- 11
- Forks
- 10
- Avg merge
- 14h 18m
- Merged PRs (30d)
- 1
Description
### How will this affect the next version number?
New feature (increment MINOR version)
### Is your feature request related to a problem?
Yes, @chengzhuzhang noted [here](https://github.com/E3SM-Project/zstash/pull/427#issuecomment-4615985477):
> There have been complaints about zstash check being slow.
Please note that this issue is similar, but not identical to #410. That issue was closed because the `--tars` option was a sufficient solution to the problem it considered:
> We need zstash check to be able to immediately jump to a specified point
That is, that issue was resolved by running on fewer tars. This issue asks if we can get more tars-per-time-unit.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
Once #427 is merged, profiling should be added for `zstash check` as well. A few important considerations for that:
1. The existing performance records don't have `check` data, so we will need to allow for backwards compatibility in the performance scripts.
2. #427 already produces 5 relatively busy plots just for `create`, `update`, and `extract` (both sequential and parallel), so we will need to think about the best way to visualize `check`. Include it on the same output? A separate figure? Does it make sense to compare it to how long the full `extract` took?
3. Make sure that running `zstash check` doesn't interfere with other results. Considering we're running `extract twice (sequential and parallel), I think this should be feasible.
That is, this issue should only be considered resolved once we have both:
1. A more efficient `zstash check`
2. Performance profiling scripts that confirm this is so.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.