pydata / pydata/xarray

Handling of non-string dimension names

Open
#5,148 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-error reporting
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

While working on a pull request (#5149) for #5146 I came across an inconsistency in allowed dimension names. If I try and create a DataArray with a non-string dimension, I get a TypeError:

>>> import xarray as xr
>>> da = xr.DataArray(np.ones((5, 5)), dims=[1, "y"])
...
TypeError: dimension 1 is not a string

But creating it with a string and renaming it works:

>>> da = xr.DataArray(np.ones((5, 5)), dims=["x", "y"]).rename(x=1)
>>> da
<xarray.DataArray (1: 5, y: 5)>
array([[1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.]])
Dimensions without coordinates: 1, y

I can create a dataset via this renaming, but trying to get the repr value fails as xarray.core.utils.SortedKeysDict tries to sort it and cannot compare the string dimension to the int dimension:

>>> import xarray as xr
>>> ds = xr.Dataset({"test": xr.DataArray(np.ones((5, 5)), dims=["x", "y"]).rename(x=1)})
>>> ds
...
~/software/external/xarray/xarray/core/formatting.py in dataset_repr(ds)
    519 
    520     dims_start = pretty_print("Dimensions:", col_width)
--> 521     summary.append("{}({})".format(dims_start, dim_summary(ds)))
    522 
    523     if ds.coords:

~/software/external/xarray/xarray/core/formatting.py in dim_summary(obj)
    422 
    423 def dim_summary(obj):
--> 424     elements = [f"{k}: {v}" for k, v in obj.sizes.items()]
    425     return ", ".join(elements)
    426 

~/software/external/xarray/xarray/core/formatting.py in <listcomp>(.0)
    422 
    423 def dim_summary(obj):
--> 424     elements = [f"{k}: {v}" for k, v in obj.sizes.items()]
    425     return ", ".join(elements)
    426 

/usr/lib/python3.9/_collections_abc.py in __iter__(self)
    847 
    848     def __iter__(self):
--> 849         for key in self._mapping:
    850             yield (key, self._mapping[key])
    851 

~/software/external/xarray/xarray/core/utils.py in __iter__(self)
    437 
    438     def __iter__(self) -> Iterator[K]:
--> 439         return iter(self.mapping)
    440 
    441     def __len__(self) -> int:

~/software/external/xarray/xarray/core/utils.py in __iter__(self)
    504     def __iter__(self) -> Iterator[K]:
    505         # see #4571 for the reason of the type ignore
--> 506         return iter(sorted(self.mapping))  # type: ignore[type-var]
    507 
    508     def __len__(self) -> int:

TypeError: '<' not supported between instances of 'str' and 'int'

The same thing happens if I call rename on the dataset rather than the array it is initialised with.

If the initialiser requires the dimension names to be strings, and other code (which includes the HTML formatter I was looking at when I found this) assume that they are, then rename and any other method which can alter dimension names should also enforce the string requirement.

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: 851d85b9203b49039237b447b3707b270d613db5
python: 3.9.2 (default, Feb 20 2021, 18:40:11)
[GCC 10.2.0]
python-bits: 64
OS: Linux
OS-release: 5.11.13-arch1-1
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_NZ.UTF-8
LOCALE: en_NZ.UTF-8
libhdf5: 1.12.0
libnetcdf: 4.7.4

xarray: 0.17.0
pandas: 1.2.3
numpy: 1.20.1
scipy: 1.6.2
netCDF4: 1.5.6
pydap: None
h5netcdf: 0.10.0
h5py: 3.2.1
Nio: None
zarr: None
cftime: 1.4.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.2.2
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.03.0
distributed: 2021.03.0
matplotlib: 3.4.1
cartopy: 0.18.0
seaborn: 0.11.1
numbagg: None
pint: None
setuptools: 54.2.0
pip: 20.3.1
conda: None
pytest: 6.2.3
IPython: 7.22.0
sphinx: 3.5.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dimension validation in the DataArray initializer and the rename paths. Inspect formatting.py's dim_summary and utils.py's SortedKeysDict, which currently fail when string and integer dimension names are mixed. Done means dimension names are handled consistently: invalid non-string names are rejected by rename paths, and dataset repr no longer fails for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.