pydata / pydata/xarray

Update dataset attributes and sync to disk store

Open
#3,536 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Summary: I would like to be able to update dataset/dataarray attributes and sync the updates to disk.

Regarding the Zarr backend in Xarray, at the moment once datasets are read from disk all modifications are in memory only. This means that for the particular case of metadata stored as attributes, in order to update an attribute on disk there are two choices:

  • Assign the new attribute to the dataset and write the whole dataset again to disk.
  • Use Zarr directly to access the dataset attributes and perform an 'attribute sync'.

The former is prohibitive , in particular for large datasets, but in general it is not a good practice that in order to modify one attribute I need to rewrite the whole pixel data.

The second is possible, and certainly I could imagine the need to write a custom function to do this. Is that something that should be available in the xarray library though? Something like ds.sync_attributes() that updates the attributes on disk?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names Xarray's Zarr backend and a proposed ds.sync_attributes() entry point, but no file or test. Start by locating the Zarr backend handling for dataset and dataarray attributes and comparing it with Zarr's attribute-sync behavior. Done means defining and implementing a supported way to persist attribute changes without rewriting pixel data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.