fsspec / fsspec/kerchunk

Non-deterministic behaviour of MultiZarrToZarr

Open
#434 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I'm seeing what looks like non-deterministic behaviour with MultiZarrToZarr. It happens both with and without writing to parquet. Sometimes I get the expected result written, sometimes I don't. See the standalone notebook here.

I'm also seeing different behaviour when I do/don't write out to parquet. Even the act of writing to parquet is changing the json returned by MultiZarrToZarr.

(The example in the notebook comes from a snippet I created whilst trying to work around the bug reported in #386 - see https://github.com/fsspec/kerchunk/issues/386#issuecomment-1979532213. It's slightly odd because I preprocess then postprocess in the same MultiZarrToZarr call. But that should be a perfectly well-defined thing to do.)

This behaviour might be specific to the application of preprocess and postprocess, but it also might be indicative of a more wide-reaching bug.

I am hoping I am somehow using MultiZarrToZarr incorrectly, but given how I'm running identical code in consecutive cells and seeing different behaviour, I don't understand how this can be user error.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the standalone notebook linked in the issue and reproduce the consecutive MultiZarrToZarr calls, both with and without parquet output. Compare the returned JSON and generated results when using preprocess followed by postprocess; done means identical code produces deterministic, expected output in both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.