Non-deterministic behaviour of MultiZarrToZarr
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm seeing what looks like non-deterministic behaviour with MultiZarrToZarr. It happens both with and without writing to parquet. Sometimes I get the expected result written, sometimes I don't. See the standalone notebook here.
I'm also seeing different behaviour when I do/don't write out to parquet. Even the act of writing to parquet is changing the json returned by MultiZarrToZarr.
(The example in the notebook comes from a snippet I created whilst trying to work around the bug reported in #386 - see https://github.com/fsspec/kerchunk/issues/386#issuecomment-1979532213. It's slightly odd because I preprocess then postprocess in the same MultiZarrToZarr call. But that should be a perfectly well-defined thing to do.)
This behaviour might be specific to the application of preprocess and postprocess, but it also might be indicative of a more wide-reaching bug.
I am hoping I am somehow using MultiZarrToZarr incorrectly, but given how I'm running identical code in consecutive cells and seeing different behaviour, I don't understand how this can be user error.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the standalone notebook linked in the issue and reproduce the consecutive MultiZarrToZarr calls, both with and without parquet output. Compare the returned JSON and generated results when using preprocess followed by postprocess; done means identical code produces deterministic, expected output in both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100