pydata / pydata/xarray

automatic handling of concurrent connections limit

Open
#4,459 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-documentation
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

I'm getting errors when I try to open 48 IMERG time steps through opendap. I don't think this is a bug, but it's not clear how I can configure xarray to avoid it. The server doesn't want to be overwhelmed by multiple data layer requests, which is reasonable, but I'm not asking for big data here. It would be fast enough if we downloaded the data one time step at a time.

urls = [build_IMERG_url(half_hour, run, version) for half_hour in half_hours]
imerg = xr.open_mfdataset(urls) # Reaches the concurrent connection limit
precipitation = imerg['precipitationCal']
daily = precipitation.mean('time') * 24

Can xarray avoid this error automatically? If not, is there a way to limit my creation of concurrent connections?

I don't know if using the netcdf library directly avoids this issue. I did try to write a for loop to connect to opendap 1 at a time with xr.open_dataset(), but if still gave the concurrent connections error. Obviously, I could just download all the data files instead of using opendap to subset, but this is not the desired outcome.

Example URL:
https://gpm1.gesdisc.eosdis.nasa.gov:443/opendap/GPM_L3/GPM_3IMERGHHL.06/2020/260/3B-HHR-L.MS.MRG.3IMERG.20200916-S000000-E002959.0000.V06B.HDF5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the concurrent-connection error with xr.open_mfdataset() on the listed IMERG URLs, then compare it with the xr.open_dataset() loop described in the issue. Trace the open_mfdataset entry point and its remote dataset handling. Done means users can limit or avoid concurrent OpenDAP connections without downloading all files, with coverage for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.