gridftp
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Hello,
We are trying to use a small subset of CMIP6 data from ESGF server.
They expose their NetCDF files in different ways
from pyesgf.search import SearchConnection
server='https://esgf-data.dkrz.de/esg-search'
conn = SearchConnection(server, distrib=True)
source_id='CMCC-CM2-HR4'
activity_id='OMIP'
experiment_id='omip2'
variable_id='vmo'
ctx = conn.new_context(
project='CMIP6',
source_id=source_id,
experiment_id=experiment_id,
variable=variable_id,
frequency='mon',
)
result = ctx.search()[0]
files = result.file_context().search()
files[35].urls
Which gives
defaultdict(list,
{'HTTPServer': [('http://esgf-node2.cmcc.it/thredds/fileServer/esg_dataroot/CMIP6/OMIP/CMCC/CMCC-CM2-HR4/omip2/r1i1p1f1/Omon/vmo/gn/v20200226/vmo_Omon_CMCC-CM2-HR4_omip2_r1i1p1f1_gn_200801-201812.nc',
'application/netcdf')],
'GridFTP': [('gsiftp://esgf-node2.cmcc.it:2811//esg_dataroot/CMIP6/OMIP/CMCC/CMCC-CM2-HR4/omip2/r1i1p1f1/Omon/vmo/gn/v20200226/vmo_Omon_CMCC-CM2-HR4_omip2_r1i1p1f1_gn_200801-201812.nc',
'application/gridftp')],
'OPENDAP': [('http://esgf-node2.cmcc.it/thredds/dodsC/esg_dataroot/CMIP6/OMIP/CMCC/CMCC-CM2-HR4/omip2/r1i1p1f1/Omon/vmo/gn/v20200226/vmo_Omon_CMCC-CM2-HR4_omip2_r1i1p1f1_gn_200801-201812.nc.html',
'application/opendap-html')],
'Globus': [('globus:4101e3a0-b7df-11eb-a16a-5fad80e6400b/esg_dataroot/CMIP6/OMIP/CMCC/CMCC-CM2-HR4/omip2/r1i1p1f1/Omon/vmo/gn/v20200226/vmo_Omon_CMCC-CM2-HR4_omip2_r1i1p1f1_gn_200801-201812.nc',
'Globus')]})
We just need small subset of netcdf file, and I would like to make kerchunk catalogue of it. I can use the HTTPServer link to transform it to kerchunk catalogue, but just out of curiosity, can it also handle 'ftp' or 'open dap' or 'gridftp' ?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pyesgf SearchConnection example in the issue and reproduce the result.file_context().search() output, including the HTTPServer, OPENDAP, GridFTP, and Globus URLs. Review kerchunk's existing URL and protocol handling to determine which forms are supported and what work would be needed for the others; done means a confirmed support status with tests or documented limitations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100