astropy / astropy/pyvo

Queries to IRSA TAP failing intermittently

Open
#629 1 comment 0 reactions 0 assignees View on GitHub
bug component: dal component: TAP
Dominant language
Python
Stars
88
Forks
64
Avg merge
38m
Merged PRs (30d)
1

Description

While trying to pull data from IRSA via TAP, I get the following error:

```
Traceback (most recent call last):
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connection.py", line 464, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 1378, in getresponse
response.begin()
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connection.py", line 464, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 1378, in getresponse
response.begin()
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/query.py", line 258, in execute_votable
return votableparse(self.execute_stream(post=post).read)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 1101, in execute_stream
return super().execute_stream(post=post)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/utils/decorators.py", line 9, in wrapper
raw = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/query.py", line 207, in execute_stream
response = self.submit(post=post)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 1134, in submit
response = self._session.post(
^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/adapters.py", line 682, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/fedde/NEOSurveyor/NEOSRef/scripts/make_neosref.py", line 18, in
catwise = get_catwise(ra,dec)
^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/NEOSurveyor/NEOSRef/scripts/get_catwise.py", line 27, in get_catwise
catwise = Irsa.query_tap(query=query).to_table()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/astroquery/ipac/irsa/core.py", line 78, in query_tap
return self.tap.search(query, language='ADQL', maxrec=maxrec)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 278, in run_sync
**keywords).execute()
^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 1117, in execute
return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/query.py", line 261, in execute_votable
raise DALFormatError(e, self.queryurl)
pyvo.dal.exceptions.DALFormatError: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
```

The error is intermittent, but happens often enough to be a nuisance. The query I was trying to run is:

```
columns = 'source_id,ra,sigra,dec,sigdec,sigradec,pmra,sigpmra,pmdec,sigpmdec,w1mpro,w1sigmpro,w1snr,w1rchi2,w1sat,w2mpro,w2sigmpro,w2snr,w2rchi2,w2sat'

query = ("select "+columns+" from catwise_2020 where ra>="+str(minra)+" and ra<"+str(maxra)+" and dec>="+str(mindec)+" and dec<"+str(maxdec)+" and nb=1 and ab_flags='00' and cc_flags='0000'")`

catwise = Irsa.query_tap(query=query).to_table()
```
Where minra,maxra,mindec,maxdec are set iteratively within a for loop. Entering debug mode and doing `votableparse(self.execute_stream(post=post).read)` returns a valid VO table ``, which is very puzzling.

The code never seems to fail twice on the same set of minra,maxra,mindec,maxdec, so I would think that getting astroquery to retry instead of crashing would fix the issue.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pyvo/dal/tap.py and the TAP query execution path shown in the traceback, especially submit and run_sync. Reproduce the intermittent RemoteDisconnected failure against the IRSA TAP service, then verify that a transient failed request is retried without breaking successful VOTable parsing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.