Explain target_protocol and remote_protocol
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm trying to understand fsspec-reference-maker better.
Consider the following code from the pangeo-forge hdf-reference tutorial:
m = fsspec.get_mapper(
"reference://",
fo=ref_url,
target_protocol="file",
remote_protocol="s3",
skip_instance_cache=True,
)
Please help me understand the need for target_protocol and remote_protocol
target_protocol
AFAICT, this is the name of the protocol needed to open ref_url. Is this always needed? What if the protocol is already in ref_url, e.g. `ref_url = "https://..."? What if the two are inconsistent?
Why don't we just infer the target_protocol from ref_url?
remote_protocl
AFAICT, this is the protocol used for opening the underlying reference files. But that is already encoded in the reference file!!. Here, the beginning of ref_url is
{"version": 1, "templates": {"a": "s3://esgf-world/CMIP6/OMIP/NOAA-GFDL/GFDL-CM4/omip1/r1i1p1f1/Omon/thetao/gr/v20180701/thetao_Omon_GFDL-CM4_omip1_r1i1p1f1_gr_170801-172712.nc", "b": "s3://esgf-world/CMIP6/OMIP/NOAA-GFDL/GFDL-CM4/omip1/r1i1p1f1/Omon/thetao/gr/v20180701/thetao_Omon_GFDL-CM4_omip1_r1i1p1f1_gr_172801-174712.nc", "c": "s3://esgf-world/CMIP6/OMIP/NOAA-GFDL/GFDL-CM4/omip1/r1i1p1f1/Omon/thetao/gr/v20180701/thetao_Omon_GFDL-CM4_omip1_r1i1p1f1_gr_174801-176712.nc", "d": "s3://esgf-world/CMIP6/OMIP/NOAA-GFDL/GFDL-CM4/om ...
All of those s3://s mean that the data should be read with s3 protocol. So why do we also need to specify remote_protocol? What if I put remote_protocol='gcs' but the actual references are s3? Wouldn't this cause problems?
In summary, it feels to me like both these specifiers are redundant and therefore a source of potential bugs. But I'm sure I'm missing something.
In any case, the documentation on these options could be improved.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked ReferenceFileSystem API documentation and the pangeo-forge HDF reference tutorial example. Clarify what target_protocol and remote_protocol control, whether they are required, and how conflicting values behave; the documentation is done when these questions are answered with accurate guidance and examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100