scverse / scverse/anndata

Consolidated Metadata Handling

Open
#2,016 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

backend: zarr topic: backed topic: cloud ☁️ topic: io
Dominant language
Python
Stars
773
Forks
209
Avg merge
19h 41m
Merged PRs (30d)
32

Description

Please describe your wishes and possible alternatives to achieve the desired result.

anndata.io.write_zarr for 0.12 onwards writes consolidated metadata by default. I think this is a good thing for cloud users, but creates consistency issues. Luckily, out write format has the advantage that you need to use our functions specifically to write correctly so we can introcuce a check as in #2015 to handle consolidated metadata.

I am creating this issue to feel out how best to allow users to opt-out of consolidated metadata. The options might be

  1. A new argument to write_zarr
  2. A setting
  3. A clearly stated way of creating a shim class that overrides the property introduced in https://github.com/zarr-developers/zarr-python/pull/3119 (and then for v2, potentially do nothing thus encouraging people to upgrade).
  4. Do nothing (which is similar to option 3) and trust that our handling is good enough

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the anndata.io.write_zarr entry point and the consistency check discussed in issue #2015. Review the Zarr property change linked in the issue and compare the four proposed approaches for opting out of consolidated metadata. Done means a decided, documented approach with its expected behavior for supported versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.