pydata / pydata/xarray

to_zarr append errors when using encoding point in wrong direction

Open
#5,263 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-documentation topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

When I try to append to a zarr store created from encoded data it isn't clear that I need to decode the data I want to append.

When I run the attached code, I get an error:

ValueError: failed to prevent overwriting existing key add_offset in attrs. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.

Following that error, if I just remove the encoding I can append to the zarr store, but the data isn't encoding correctly.

The fix is to xr.decode_cf(data) and then append.

It would help if the to_zarr append documentation made this clearer.

Example given here:
https://github.com/cgentemann/cloud_science/blob/master/make_zarr/zarr_append_issue.ipynb

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked zarr_append_issue.ipynb to understand the reported append workflow, then find the xarray documentation for to_zarr append behavior. Clarify that data should be decoded with xr.decode_cf(data) before appending, and confirm that the documented guidance addresses the encoding error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.