MultiZarrToZarr does not pass `target_options` to the reference filesystems
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Hi there! I was trying to get MultiZarrToZarr to work on a set of files that have been gzip compressed. Fsspec's fsspec.open function has a compression parameter that can be passed to tell fsspec to decompress the files on the fly. I have gotten this to work using zarr files by doing something like this:
fsspec.filesystem(
"reference",
fo="s3://bucket/my-compressed-zarr-file.json.gz",
target_options={'compression': 'gzip'},
remote_protocol='s3',
remote_options={
'anon': False,
},
)
I was hoping that this would also work with MultiZarrToZarr and be equally as simple as passing the target_options, however, it seems that while MultiZarrToZarr opens the files using the target_options here to get the file name:
https://github.com/fsspec/kerchunk/blob/dc66b2cd85ce170fbc0fbc652cc80f54439bd786/kerchunk/combine.py#L265
It then does not pass the target options to the fsspec.filesystem call here:
Is this a bug or is there some reason why the target_options can't be passed to fsspec.filesystem there?
Analysis of Compressed files
From what I can tell by running this with a debugger, this is what's happening:
- The call to
open_fileshere succeeds:
https://github.com/fsspec/kerchunk/blob/dc66b2cd85ce170fbc0fbc652cc80f54439bd786/kerchunk/combine.py#L265 - The call to
fs.catdoes not include thetarget_optionsso the file is not decompressed when read:
https://github.com/fsspec/kerchunk/blob/dc66b2cd85ce170fbc0fbc652cc80f54439bd786/kerchunk/combine.py#L270 - The json decoding fails because it's trying to decode the compressed data and the
fo_listis set to the original list of filenames.
https://github.com/fsspec/kerchunk/blob/dc66b2cd85ce170fbc0fbc652cc80f54439bd786/kerchunk/combine.py#L274 - The
fsspec.filesystemcall opens the files again, but since thetarget_optionsare not passed in, the data is not decompressed and the json decoding fails:
https://github.com/fsspec/kerchunk/blob/dc66b2cd85ce170fbc0fbc652cc80f54439bd786/kerchunk/combine.py#L277
Solution
I think there are 2 additional places that the target options need to be passed in
In thefs.catcall via the**kwargsparameter:
https://github.com/fsspec/kerchunk/blob/dc66b2cd85ce170fbc0fbc652cc80f54439bd786/kerchunk/combine.py#L270
API Docs for fs.cat: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem
Source code for fs.cat: https://github.com/fsspec/filesystem_spec/blob/4517882f67d635d50b54cd53fd04ee3a37b6943c/fsspec/spec.py#L844
EDIT: After trying this out, it seems that the s3fs implementation for cat_file doesn't work the way the syncronous abstract class does where the kwargs are passed to a call to fs.open. The s3fs _cat_file doesn't support kwargs at all: https://github.com/fsspec/s3fs/blob/f3f63cbfbfe71a4355abd63cafd8c678c4a5a0af/s3fs/core.py#L1113
- In the
fs.filesystemcall:
https://github.com/fsspec/kerchunk/blob/dc66b2cd85ce170fbc0fbc652cc80f54439bd786/kerchunk/combine.py#L277
Workaround
I believe I can work around this by opening the files myself and passing in the zarr dictionaries directly. It's just more code for me to write :)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in kerchunk/combine.py around the open_files call at line 265 and the fsspec.filesystem call at lines 277-283, then review the described fs.cat behavior and s3fs cat_file limitation. Ensure MultiZarrToZarr handles target_options when reading compressed reference files and confirm that JSON decoding succeeds for the reported gzip case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100