Getting DataArrays from netCDF4 files correctly and without hassle
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Context
Consider a netCDF4 file with a group structure. For example, the following toy:
import netCDF4 as nc
# netCDF4 file
f = nc.Dataset('simple_hierarchy.nc', 'w')
# coordinates in root
f.createDimension('x', 3)
f.createVariable('x', 'f4', ('x',), fill_value=False)
f['x'][:] = [1.1, 2.2, 3.3]
f.createDimension('y', 2)
f.createVariable('y', 'f4', ('y',), fill_value=False)
f['y'][:] = [-0.9, -1.8]
# variables in root
f.createVariable('u', 'i1', (), fill_value=False)
f.createVariable('v', 'u1', ('x','y'), fill_value=False)
# group
f.createGroup('g')
g = f['g']
# new/modified coordinates in g
g.createDimension('y', 3)
g.createVariable('y', 'f4', ('y',), fill_value=False)
g['y'][:] = [-0.9, -1.8, -2.7]
# variable in g
g.createVariable('w', 'u1', ('x', 'y'), fill_value=False)
f.close()
Current behavior
-
It is currently a hassle to get a DataArray from variable in a group with multiple non-coordinate variables:
>>> xr.open_dataarray('simple_hierarchy.nc') … ValueError: Given file dataset contains more than one data variable. Please read with xarray.open_dataset and then select the variable you want. >>> xr.open_dataarray('simple_hierarchy.nc', group='v') xr.open_dataarray('simple_hierarchy.nc', group='v') … OSError: [Errno group not found: v] 'v' >>> xr.open_dataarray('simple_hierarchy.nc', drop_variables='u') <xarray.DataArray 'v' (x: 3, y: 2)> array([[120, 219], [178, 172], [ 9, 127]], dtype=uint8) Coordinates: * x (x) float32 1.1 2.2 3.3 * y (y) float32 -0.9 -1.8 -
Also, coordinates defined at a group level closer tot the root are not taken into account:
>>> xr.open_dataarray('simple_hierarchy.nc', group='g') <xarray.DataArray 'w' (x: 3, y: 3)> array([[216, 219, 178], [172, 9, 127], [ 0, 0, 64]], dtype=uint8) Coordinates: * y (y) float32 -0.9 -1.8 -2.7 Dimensions without coordinates: xSo the DataArray is not loaded correctly, as part of its defining coordinates are missing.
Suggested behavior
-
Add a
variablekwarg in theopen_dataarraymethod:>>> xr.open_dataarray('simple_hierarchy.nc', variable='v') <xarray.DataArray 'v' (x: 3, y: 2)> array([[120, 219], [178, 172], [ 9, 127]], dtype=uint8) Coordinates: * x (x) float32 1.1 2.2 3.3 * y (y) float32 -0.9 -1.8 -
Have the function that loads variables go up the group hierarchy to see if some coordinate arrays can be found for dimensions lacking them within this group:
>>> xr.open_dataarray('simple_hierarchy.nc', group='g') <xarray.DataArray 'w' (x: 3, y: 3)> array([[216, 219, 178], [172, 9, 127], [ 0, 0, 64]], dtype=uint8) Coordinates: * x (x) float32 1.1 2.2 3.3 * y (y) float32 -0.9 -1.8 -2.7I guess care needs to be taken as well upon writing to netCDF, to make sure no spurious dimension/coordinate definitions are added.
Version
xarray 0.9.6
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the open_dataarray entry point and reproduce the toy netCDF4 hierarchy described in the issue. Implement the requested variable selection and parent-group coordinate lookup, then verify that grouped variables load with all defining coordinates and that existing behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100