MetOffice / MetOffice/ANTS

Unsafe casting in netCDF save

Open
#67 0 comments 0 reactions 0 assignees View on GitHub
➡ medium priority 🐜 bug
Dominant language
Python
Stars
7
Forks
10
Avg merge
1d 19h
Merged PRs (30d)
5

Description

# 🐜 Bug Report

The dtype casting for the netCDF4-classic save is incorrect. For example, `uint32` and `uint64` are being cast to `i8` (i.e. long int, `int64`), but netCDF4-classic does not permit long integers.

The list of permitted data types can be found [​here](https://docs.unidata.ucar.edu/nug/current/md_types.html). Note the remark:

All the unsigned ints and the 64-bit ints are for CDF5 or netCDF-4 files only.

I suggest the best approach is to not attempt to cast as part of ANTS, but leave it to the underlying iris/netCDF libraries where possible. The only case where casting should be done is for boolean data, where the data should be cast to `i1` (i.e. "byte") with a valid_range attribute of `[0, 1]` added. Any other dtype casting should be the responsibility of the science application, not core ANTS code.

## Version
This bug exists in ANTS 2.2

## Additional Context

Numpy provides a `can_cast` method that is used in netCDF saving in ANTS:

This is generating a FutureWarning from dask:

FutureWarning: The numpy.can_cast function is not implemented by Dask array. You may want to use the da.map_blocks function or something similar to silence this warning. Your code may stop working in a future release.

Contributor guide

Open the contributing guide

Research direction

Start by locating ANTS's netCDF4-classic save entry point and the numpy.can_cast usage mentioned in the report. Check how unsigned and 64-bit dtypes are handled, then verify that only booleans are cast to i1 with a valid_range of [0, 1] while other casting is left to the iris/netCDF libraries.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.