pydata / pydata/xarray

Suggestion: Add option for default_fillvals to open_dataset

Open
#2,374 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contrib-help-wanted topic-CF conventions
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Hi,

May I suggest having a default_fillvals option to xarray.open_dataset (and xarray.open_dataarray)?

My problem:

I have netcdf data containing flagged data, that is flagged with the netcdf default fill value of 9.96...e+36. But xarray (0.10.8) only masks arrays that have an explicit fill_value set:

import netCDF4, xarray, numpy

nc = netCDF4.Dataset('test.nc', 'w', format='NETCDF4')
nc.createDimension('x', 3)

var1 = nc.createVariable('var1', 'f8', ('x',))
var2 = nc.createVariable('var2', 'f8', ('x',), fill_value=netCDF4.default_fillvals['f8'])

var1[:] = numpy.array([0., 1., netCDF4.default_fillvals['f8']])
var2[:] = numpy.array([0., 1., netCDF4.default_fillvals['f8']])
print('netCDF4 var1', nc.variables['var1'][:])
print('netCDF4 var2', nc.variables['var2'][:])
nc.close()

ds = xarray.open_dataset('test.nc')
print('xarray var1', ds.var1[:])
print('xarray var2', ds.var2[:])

The problem is, that ds.var1 and ds.var2 are interpreted differently, although netCDF4 shows both as masked:

netCDF4 var1 [0.0 1.0 --]
netCDF4 var2 [0.0 1.0 --]
xarray var1 <xarray.DataArray 'var1' (x: 3)>
array([0.00000e+00, 1.00000e+00, 9.96921e+36])
Dimensions without coordinates: x
xarray var2 <xarray.DataArray 'var2' (x: 3)>
array([ 0.,  1., nan])
Dimensions without coordinates: x

I agree, that it is a good default, to mask data, only if the fill_value attribute is set. But I think it would be useful to be able to pass default_fill values to open_dataset to enable reading data, that uses the implicit default values.

What do you think?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the open_dataset and open_dataarray entry points and reproduce the supplied netCDF4 example to understand the differing fill-value behavior. Add an option for supplying default fill values, and verify that datasets using implicit netCDF defaults are masked consistently with explicitly configured fill values.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
backend-api-design, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.