MetOffice / MetOffice/CDDS

Check packing on variables that involve arithmetoc operations

Open
#554 0 comments 0 reactions 0 assignees View on GitHub
migrated
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 15h
Merged PRs (30d)
23

Description

Migrated Jira Issue CDDSO-59
Jira creation date 21/04/2022 15:23
-------
If data is packed then netCDF compression is believed to work more efficiently (https://doi.org/10.5281/zenodo.3961931). The quantization introduced by the PP packing may be lost if the expression for a variable includes an arithmetic operation, for example `expression = 0.5 * m01s02i398 / m01s02i313` in `reffclic` may not compress as well as `expression = m01s02i208` in `rldscs`.It may be worth examining the compression ratios achieved in a variety of variables to see if this problem does exist. If it does, then it should be considered if it is worth while applying the packing again before saving as netCDF.
-------

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the handling of the reffclic and rldscs expressions and the netCDF output path. Compare compression ratios for variables with arithmetic expressions against directly packed variables, then determine whether repacking before saving improves results. Done means the comparison is documented and any resulting implementation scope is agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.