pydata / pydata/xarray

Using a forward slash in a node names of a `DataTree` causes a `RecursionError`

Open
#9,978 2 comments 1 reaction 1 assignee View on GitHub

@TomNicholas is already working on this.

Since Jan 23, 2025.

bug topic-DataTree
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

When creating an xarray.DataTree object, passing the optional children parameter with / in a key of the dictionary produces an infinite recursion loop.

It appears that a ValueError: node names cannot contain forward slashes is raised, but the program then gets stuck in a mutual recursive loop between

child._set_parent(new_parent=self, child_name=name) # xarray/core/treenode.py line 333

and

self._attach(new_parent, child_name) # xarray/core/treenode.py line 115

This happens before the ValueError can be handled by the call to xarray.DataTree.

What did you expect to happen?

Either of the following:

  • Just raise ValueError("node names cannot contain forward slashes"), without the recursion issue.
  • Even better: using / in the keys of the children parameter produces nested DataTree objects. This was be behaviour in xarray-contrib/datatree.
Minimal Complete Verifiable Example
from xarray import DataTree

# Wrapping this in try, except handles the RecursionError, not the ValueError
dt = DataTree(children={'a/b': DataTree()})
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
Traceback (most recent call last):
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 182, in children
    child._set_parent(new_parent=self, child_name=name)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 115, in _set_parent
    self._attach(new_parent, child_name)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 159, in _attach
    self._post_attach(parent, child_name)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 723, in _post_attach
    _validate_name(name)  # is this check redundant?
    ~~~~~~~~~~~~~~^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 676, in _validate_name
    raise ValueError("node names cannot contain forward slashes")
ValueError: node names cannot contain forward slashes

During handling of the above exception, another exception occurred:

... many repetitions of 'ValueError: node names cannot contain forward slashes' and it's stack trace

Traceback (most recent call last):
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 182, in children
    child._set_parent(new_parent=self, child_name=name)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 115, in _set_parent
    self._attach(new_parent, child_name)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 152, in _attach
    self._pre_attach(parent, child_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 528, in _pre_attach
    check_alignment(path, node_ds, parent_ds, self.children)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 149, in check_alignment
    align(node_ds, parent_ds, join="exact", copy=False)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 883, in align
    aligner.align()
    ~~~~~~~~~~~~~^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 572, in align
    self.find_matching_indexes()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 254, in find_matching_indexes
    obj_indexes, obj_index_vars = self._normalize_indexes(obj.xindexes)
                                  ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 215, in _normalize_indexes
    for idx, index_vars in Indexes(xr_indexes, xr_variables).group_by_index():
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File ".venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 1624, in group_by_index
    for i, index in self._id_index.items():
                    ^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 1515, in _id_index
    self.__id_index = {id(idx): idx for idx in self.get_unique()}
                                               ~~~~~~~~~~~~~~~^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "xarray_datatree_recursion_loop.py", line 2, in <module>
    DataTree(children={'a/b': DataTree(name='leaf')})
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 506, in __init__
    super().__init__(name=name, children=children)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 691, in __init__
    super().__init__(children=children)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 87, in __init__
    self.children = {name: child.copy() for name, child in children.items()}
    ^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/common.py", line 333, in __setattr__
    object.__setattr__(self, name, value)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 187, in children
    self.children = old_children
    ^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/common.py", line 333, in __setattr__
    object.__setattr__(self, name, value)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 187, in children
    self.children = old_children
    ^^^^^^^^^^^^^

... many repetitions jumping between the previous two calls

  File ".venv/lib/python3.13/site-packages/xarray/core/common.py", line 333, in __setattr__
    object.__setattr__(self, name, value)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 187, in children
    self.children = old_children
    ^^^^^^^^^^^^^

  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 115, in _set_parent
    self._attach(new_parent, child_name)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 152, in _attach
    self._pre_attach(parent, child_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 521, in _pre_attach
    if name in parent.dataset.variables:
               ^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 614, in dataset
    return self._to_dataset_view(rebuild_dims=True, inherit=True)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 553, in _to_dataset_view
    coord_vars = self._coord_variables if inherit else self._node_coord_variables
                 ^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 541, in _coord_variables
    *(p._node_coord_variables_with_index for p in self.parents),
                                                  ^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 338, in parents
    return tuple(self._iter_parents())
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 305, in _iter_parents
    node: Tree | None = self.parent
                        ^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded
Anything else we need to know?

I tested this on the latest main branch in a new conda environment, but it appears the issue is present in every version of xarray since the introduction of DataTree.

Environment

INSTALLED VERSIONS

commit: None
python: 3.13.1 (main, Dec 4 2024, 18:05:56) [GCC 14.2.1 20240910]
python-bits: 64
OS: Linux
OS-release: 6.12.9-arch1-1
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development

xarray: 2025.1.1
pandas: 2.2.3
numpy: 2.2.1
scipy: 1.15.1
netCDF4: 1.7.2
pydap: None
h5netcdf: None
h5py: None
zarr: None
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.10.0
cartopy: None
seaborn: None
numbagg: None
fsspec: None
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: None
pip: None
conda: None
pytest: 8.3.4
mypy: None
IPython: None
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.