astropy / astropy/astroquery

BUG: utils.tap hardwire 2000 limit for tap queries

Open
#2,600 3 comments 0 reactions 0 assignees View on GitHub
bug utils.tap
Dominant language
Python
Stars
791
Forks
451
Avg merge
1d 3h
Merged PRs (30d)
4

Description

The hardwired 2000 should be eliminated from the code (or to be set in a consistent or configurable way). Currently what I see is quite inconsistent, e.g. I don't see if it's hard limit why it's not being enforced for the second case.

See line https://github.com/astropy/astroquery/blob/main/astroquery/utils/tap/core.py#L272

```
>>> from astroquery.esa.xmm_newton import XMMNewton
>>> query = "select * from v_public_observations"

>>> len(XMMNewton._tap.launch_job(query).get_results())
2000

>>> len(XMMNewton._tap.launch_job(query, maxrec=300).get_results())
300

>>> len(XMMNewton._tap.launch_job(query, maxrec=3500).get_results())
2000

>>> query1 = "select TOP 3000 * from v_public_observations"

>>> len(XMMNewton._tap.launch_job(query1).get_results())
3000

>>> len(XMMNewton._tap.launch_job(query1, maxrec=300).get_results())
300

>>> len(XMMNewton._tap.launch_job(query1, maxrec=3500).get_results())
3000
```

OTOH, looking into it with pyvo's TAP, it looks like no server-side limits are set, and therefore the query runs into the timeout (I didn't try to override that, as it already demonstrates the issue). Tracebacks are cut out for better clarity:

```
>>> import pyvo as vo
>>> query = "select * from v_public_observations"
>>> tap_service = vo.dal.TAPService("https://nxsa.esac.esa.int/tap-server/tap")

>>> print(tap_service.hardlimit)
...
DALServiceError: Hard limit not exposed by the service

>>> print(tap_service.maxrec)
...
DALServiceError: Default limit not exposed by the service

>>> tap_results = tap_service.search(query)
...
DALQueryError: Maximum execution time (60 s) reached. Job aborted.

>>> tap_results = tap_service.search(query, maxrec=100)
DALOverflowWarning: Partial result set. Potential causes MAXREC, async storage space, etc.

>>> len(tap_results)
100
```

As for the TOP limited query:

```
>>> query1 = "select TOP 2500 * from v_public_observations"
>>> len(tap_service.search(query1))
2500
>>> len(tap_service.search(query1, maxrec=300))
DALOverflowWarning: Partial result set. Potential causes MAXREC, async storage space, etc.
300
>>> len(tap_service.search(query1, maxrec=3000))
2500
```

Contributor guide

Open the contributing guide

Research direction

Start at astroquery/utils/tap/core.py line 272 and trace how launch_job applies maxrec to TAP queries, comparing the reported plain and TOP-limited examples with pyvo's TAPService behavior. Done means the hardwired 2000 behavior is removed or consistently configurable, while the documented maxrec examples behave as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
api, backend-api-design
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.