MetOffice / MetOffice/lfric_core

Honour `_FillValue` attribute when loading ancillary files

Open
#294 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Fortran
Stars
26
Forks
73
Avg merge
1d 8h
Merged PRs (30d)
4

Description

### Are there any linked Issues or Pull Requests?

- ticket:4168 - Original LFRic Trac ticket.
- [ancil:2130](https://code.metoffice.gov.uk/trac/ancil/ticket/2130) - ANCIL workaround Trac ticket.

### Brief description

Presently, the `_FillValue` attribute is ignored when LFRic reads in ancillary files (and presumably other input files). This attribute ([attribute conventions](https://docs.unidata.ucar.edu/nug/current/attribute_conventions.html)) is used to identify missing data] (e.g. to mask out land points for sea-only fields).

### Further details of the issue.

Because the `_FillValue` attribute is ignored, this imposes a constraint on any tools creating files to be read into LFRic that they need to use the model RMDI value as a `_FillValue`.

Instead, the LFRic load should recognise `_FillValue` as representing missing data and translate this to RMDI.

There is an additional complexity in that the netCDF specification (linked above) includes "It is not necessary to define your own _FillValue attribute for a variable if the default fill value for the type of the variable is adequate". So the absence of a `_FillValue` attribute does not imply there is no missing data.

The default fill values can be found most easily with python:

```python
In [1]: import netCDF4

In [2]: netCDF4.default_fillvals
Out[2]:
{'S1': '\x00',
'i1': -127,
'u1': 255,
'i2': -32767,
'u2': 65535,
'i4': -2147483647,
'u4': 4294967295,
'i8': -9223372036854775806,
'u8': 18446744073709551614,
'f4': 9.969209968386869e+36,
'f8': 9.969209968386869e+36}
```

Note that none of these defaults match the LFRic RMDI (the largest finite negative value).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No implementation files, tests, or entry points are named. Start by locating the LFRic ancillary-file loader and its netCDF attribute handling, then inspect how missing values become RMDI. Done means explicit and type-default _FillValue values are recognised and translated to RMDI, with coverage for the supported input types.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.