intake / intake/intake-esm

Intake_esm feature: Support for Kerchunk (reference) on s3 file system

Open
#603 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

Related Issue:
Intake_esm catalog hosted on an s3 file system which points to kerchunk files also hosted on the same file system cannot be accessed properly by existing intake_esm scripts.

Solution:
A small change to line 53 onwards in intake_esm/source.py:
```
if data_format == 'reference':
if 's3://' in urlpath:
import s3fs
xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = s3fs.S3FileSystem(**kwargs).open(urlpath, **kwargs)
else:
xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = urlpath
```

Alternatives:
There might be other options to consider/features in intake_esm I'm not taking advantage of, if anyone has any insight let me know.
The above change relates to a later section:
`fsspec.get_mapper('reference://', fo=ref ...)`
Where for a kerchunk file on s3, ref needs to be the S3FileSystem object not the string path.

Contributor guide

Open the contributing guide

Research direction

Start by reading intake_esm/source.py around line 53 and trace the later fsspec.get_mapper('reference://', fo=ref ...) call. Verify how reference files are opened for s3:// URLs versus local paths, then confirm that an intake_esm catalog and its kerchunk files on the same S3 filesystem can be accessed while existing local behavior still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, data
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.