Add open_mfdatatree
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
Currently we have an
open_datatreefunction which opens a single netcdf file (or zarr store). We could imagine anopen_mfdatatreefunction which is analogous toopen_mfdataset, which can open multiple files at once.As
DataTreehas a structure essentially the same as that of a filesystem, I'm imagining a use case where the user has a bunch of data files stored in nested directories, e.g.project /experimental data.nc /simulation /highres output.nc /lowres output.ncWe could look through all of these folders recursively, open any files found of the correct format, and store them in a single tree.
We could even allow for multiple data files in each folder if we called
open_mfdataseton all the files found in each folder.EDIT: We could also save a tree out to multiple folders like this using a
save_mfdatatreemethod.This might be particularly useful for users who want the benefit of a tree-like structure but are using a file format that doesn't support groups.
Originally posted by @TomNicholas in https://github.com/xarray-contrib/datatree/issues/51#issue-1082703410
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing open_datatree entry point and comparing it with open_mfdataset. Define how recursive directories, supported file formats, and multiple files per folder should map into one DataTree, then verify the chosen behavior with tests for the nested example and the issue's proposed save_mfdatatree counterpart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100