Check packing on variables that involve arithmetoc operations
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 23
Description
Migrated Jira Issue CDDSO-59
Jira creation date 21/04/2022 15:23
-------
If data is packed then netCDF compression is believed to work more efficiently (https://doi.org/10.5281/zenodo.3961931). The quantization introduced by the PP packing may be lost if the expression for a variable includes an arithmetic operation, for example `expression = 0.5 * m01s02i398 / m01s02i313` in `reffclic` may not compress as well as `expression = m01s02i208` in `rldscs`.It may be worth examining the compression ratios achieved in a variety of variables to see if this problem does exist. If it does, then it should be considered if it is worth while applying the packing again before saving as netCDF.
-------
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the handling of the reffclic and rldscs expressions and the netCDF output path. Compare compression ratios for variables with arithmetic expressions against directly packed variables, then determine whether repacking before saving improves results. Done means the comparison is documented and any resulting implementation scope is agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100