intake / intake/intake-thredds

Can I choose `parallel=True` like in `xr.open_mfdataset`?

Open
#70 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Hi! I stumbled across this `intake` driver and it looks like what I need, so thank you so much for your work!

I have already made an example for myself where I have a bunch of file locations that are on a thredds server, and then I read them in with `xr.open_mfdataset()`, like so:
```
ds = xr.open_mfdataset(filelocs, drop_variables=['siglay','siglev','Itime2'], parallel=True, compat='override',
combine='by_coords', data_vars='minimal', coords='minimal')
```
This ended up being about 2 minutes for 127 files.

But, I need to get this combined dataset represented in an `intake` catalog, which is where `intake-thredds` comes in. I think I have properly mapped the keywords I used in `xr.open_mfdataset` to the API in `intake-thredds` like this:

```
cat_url = 'https://opendap.co-ops.nos.noaa.gov/thredds/catalog/NOAA/LEOFS/MODELS/catalog.xml'
source = intake.open_thredds_merged(
cat_url, path=[date.strftime('%Y'),
date.strftime('%m'),
date.strftime('%d'),
date.strftime(f'nos.leofs.fields.????.%Y%m%d.t12z.nc')],
concat_kwargs={"dim": "time",
'data_vars': 'minimal',
'coords': 'minimal',
'compat': 'override',
'combine_attrs': "override"
},
xarray_kwargs=dict(
drop_variables=['siglay','siglev','Itime2'],
),

)
```

But, when I then try to look at the resultant combined lazily loaded Dataset with `source.to_dask()`, it takes forever to try to load and breaks with "Bad Gateway" before it can finish. The only difference I think is that I don't see how to use `parallel=True` in the call to `intake.open_thredds_merged` which I used in calling `xr.open_mfdataset`.

Is there a way to use `parallel=True`? Thank you for your help!

Contributor guide

Open the contributing guide

Research direction

Start at the intake.open_thredds_merged entry point and compare its arguments with xr.open_mfdataset, then trace source.to_dask() to see how dataset-opening options are passed through. Done means determining whether parallel=True can be supported and documenting or testing the resulting behavior, including the reported Bad Gateway case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.