pydata / pydata/xarray

Support glob patterns in `open_datatree(group=...)` for selective group loading

Open
#11,196 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Description

When working with large hierarchical datasets, users often need only a subset of groups. Currently open_datatree(group=...) accepts a single literal path to re-root the tree. This proposal extends group to accept glob patterns (e.g., */sweep_0), filtering which groups are opened without loading the entire tree first.

Use cases

Radar data (NEXRAD): Volume scan files contain dozens of sweep groups per VCP. To analyze only the lowest elevation scan across all volumes:

dt = xr.open_datatree("radar.nc", group="*/sweep_0")

Climate model output (CMIP): Multi-model archives store data in deeply nested hierarchies like /{model}/{experiment}/{variable}. To load only temperature from all models under a specific experiment:

dt = xr.open_datatree("cmip.zarr", group="*/historical/tas")

Or to compare two specific variables across all models:

dt = xr.open_datatree("cmip.zarr", group="*/historical/ta[su]")

Proposed API

When group contains glob metacharacters (*, ?, [), it switches from root-selection mode to filter mode. Matching uses the same engine as DataTree.match() (PurePosixPath.match). Root (/) and all ancestors of matched nodes are always included to form a valid tree.

Behavior summary

group value Behavior
None Load all groups (unchanged)
"VCP-34" (no glob chars) Root selection (unchanged)
"*/sweep_0" (glob chars) Filter mode — only matched groups + ancestors
Pattern matches nothing Root-only tree

Reference

PR #10742 (async DataTree open) provides the base for this work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the open_datatree API and inspect DataTree.match() for its PurePosixPath.match behavior. Review PR #10742 for the async DataTree open foundation, then verify that glob patterns load matched groups and their ancestors, while literal groups and None retain existing behavior and nonmatching patterns produce a root-only tree.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.