fsspec / fsspec/kerchunk

MultiZarrToZarr append method - coo_map not working as expected

Open
#479 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I have approximately 4000 kerchunk JSON files in the mentioned directory, all of which require MultiZarrToZarr processing to create a single reference file, and all require http calls. When I call the MultiZarrToZarr.translate() method on all of them at once, I encounter a server disconnected error. As a quick workaround, I thought of appending to the reference file in batches. However, I came across an error when attempting to append.

It seems that coo_map is not working as expected for the append operation

Additionally, I'm wondering if there's a way to append directly to an empty path, rather than first creating a reference file and then appending to it.

Could you please provide guidance on how to improve this approach?

Screenshot 2024-07-18 at 5 52 41 PM

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with MultiZarrToZarr.translate() and the append path where coo_map is handled; use the reported batch-processing case with approximately 4000 kerchunk JSON files and HTTP calls to reproduce the errors. The issue names no files or tests. Done means append handling works as expected and the supported approach for appending to an empty path is documented or implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.