Intake_esm feature: Support for Kerchunk (reference) on s3 file system
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
Related Issue:
Intake_esm catalog hosted on an s3 file system which points to kerchunk files also hosted on the same file system cannot be accessed properly by existing intake_esm scripts.
Solution:
A small change to line 53 onwards in intake_esm/source.py:
```
if data_format == 'reference':
if 's3://' in urlpath:
import s3fs
xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = s3fs.S3FileSystem(**kwargs).open(urlpath, **kwargs)
else:
xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = urlpath
```
Alternatives:
There might be other options to consider/features in intake_esm I'm not taking advantage of, if anyone has any insight let me know.
The above change relates to a later section:
`fsspec.get_mapper('reference://', fo=ref ...)`
Where for a kerchunk file on s3, ref needs to be the S3FileSystem object not the string path.
Contributor guide
Research direction
Start by reading intake_esm/source.py around line 53 and trace the later fsspec.get_mapper('reference://', fo=ref ...) call. Verify how reference files are opened for s3:// URLs versus local paths, then confirm that an intake_esm catalog and its kerchunk files on the same S3 filesystem can be accessed while existing local behavior still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100