BAAQMD / BAAQMD/InMAP-SFAB

Make NetCDF output conform to spec

Open
#8 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

* The shared InMAP NetCDF content (e.g. [ca_isrm.ncf]) appears to be invalid
* `ncvalidator` states that it fails to conform with classic CDF-2 file format spec
* It contains 5 large fixed-size variables, while the CDF-2 format allows only one
* Fresh NCO install on OS X won't read it (issue #4)
* Fixing this would be nice

## Details

Although `ncdump` gives up:

```sh
dholstius$ NC_PATH="/Users/dholstius/GitHub/BAAQMD/InMAP-SFAB/Data/GDrive/ca_isrm.nc"
dholstius$ ncdump -k $NC_PATH
ncdump: /Users/dholstius/GitHub/BAAQMD/InMAP-SFAB/Data/GDrive/ca_isrm.nc: NetCDF: One or more variable sizes violate format constraints
```

... we can fall back to reading the first four bytes:

```sh
dholstius$ od -An -c -N4 $NC_PATH
C D F 002
```

Those indicate CDF-2 (64-bit offset) format.

After `brew install pnetcdf`:

```sh
dholstius$ ncvalidator $NC_PATH
Error:
Input file contains 5 large fixed-size variables
CDF-2 format allows only one large fixed-size variable
The 1st large fixed-size variable is PrimaryPM25
The 2nd large fixed-size variable is SOA
See: http://www.unidata.ucar.edu/software/netcdf/docs/file_structure_and_performance.html#offset_format_limitations
File "/Users/dholstius/GitHub/BAAQMD/InMAP-SFAB/Data/GDrive/ca_isrm.nc" fails to conform with classic CDF-2 file format specifications
```

## Related Issues

- #4

[ca_isrm.ncf]: https://drive.google.com/drive/folders/1WmLRz7iWo2MjtSikgHEig7M0NvK2sOns?usp=sharing

Contributor guide

No contributing guide indexed for this repository

Research direction

Obtain the shared ca_isrm.nc file and run ncvalidator and ncdump to reproduce the CDF-2 errors described in the issue. Read the CDF-2 format limitations and related issue #4 first. Done means the NetCDF output conforms to the CDF-2 specification and can be read by a fresh NCO installation on OS X.

Written by the indexing model from the issue text.

Assessment

Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.