MultiZarrToZarr append method - coo_map not working as expected
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I have approximately 4000 kerchunk JSON files in the mentioned directory, all of which require MultiZarrToZarr processing to create a single reference file, and all require http calls. When I call the MultiZarrToZarr.translate() method on all of them at once, I encounter a server disconnected error. As a quick workaround, I thought of appending to the reference file in batches. However, I came across an error when attempting to append.
It seems that coo_map is not working as expected for the append operation
Additionally, I'm wondering if there's a way to append directly to an empty path, rather than first creating a reference file and then appending to it.
Could you please provide guidance on how to improve this approach?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with MultiZarrToZarr.translate() and the append path where coo_map is handled; use the reported batch-processing case with approximately 4000 kerchunk JSON files and HTTP calls to reproduce the errors. The issue names no files or tests. Done means append handling works as expected and the supported approach for appending to an empty path is documented or implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100