MultiZarrToZarr speed improvements
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Opening an issue so we can discuss processing time for MultiZarrToZarr. I'm running with the following code:
mzz = MultiZarrToZarr(
json_list,
remote_protocol='az',
remote_options={
'account_name' : 'goeseuwest'
},
xarray_open_kwargs={
'decode_cf' : False,
'mask_and_scale' : False,
'decode_times' : False,
'use_cftime' : False,
'decode_coords' : False,
},
xarray_concat_args={
"data_vars": "minimal",
"coords": "minimal",
"compat": "override",
"join": "override",
"combine_attrs": "override",
"dim": "t"
}
)
mzz.translate('combined.json')
Running mzz.translate() takes upwards of 40 minutes on a list of 144 reference jsons, which were generated with inline_threshold=300. It looks like a good chunk of the time is divided between to_zarr() and split().

I've attached the profile file here as well, which can be visualzed with snakeviz ./multizarr_profile.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with MultiZarrToZarr.translate() and profile the to_zarr() and split() paths using the attached multizarr_profile file and snakeviz. Compare the costs for 144 reference JSONs generated with inline_threshold=300, then verify that processing time is reduced without changing the combined.json output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100