BUG: utils.tap hardwire 2000 limit for tap queries
- Dominant language
- Python
- Stars
- 791
- Forks
- 451
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
The hardwired 2000 should be eliminated from the code (or to be set in a consistent or configurable way). Currently what I see is quite inconsistent, e.g. I don't see if it's hard limit why it's not being enforced for the second case.
See line https://github.com/astropy/astroquery/blob/main/astroquery/utils/tap/core.py#L272
```
>>> from astroquery.esa.xmm_newton import XMMNewton
>>> query = "select * from v_public_observations"
>>> len(XMMNewton._tap.launch_job(query).get_results())
2000
>>> len(XMMNewton._tap.launch_job(query, maxrec=300).get_results())
300
>>> len(XMMNewton._tap.launch_job(query, maxrec=3500).get_results())
2000
>>> query1 = "select TOP 3000 * from v_public_observations"
>>> len(XMMNewton._tap.launch_job(query1).get_results())
3000
>>> len(XMMNewton._tap.launch_job(query1, maxrec=300).get_results())
300
>>> len(XMMNewton._tap.launch_job(query1, maxrec=3500).get_results())
3000
```
OTOH, looking into it with pyvo's TAP, it looks like no server-side limits are set, and therefore the query runs into the timeout (I didn't try to override that, as it already demonstrates the issue). Tracebacks are cut out for better clarity:
```
>>> import pyvo as vo
>>> query = "select * from v_public_observations"
>>> tap_service = vo.dal.TAPService("https://nxsa.esac.esa.int/tap-server/tap")
>>> print(tap_service.hardlimit)
...
DALServiceError: Hard limit not exposed by the service
>>> print(tap_service.maxrec)
...
DALServiceError: Default limit not exposed by the service
>>> tap_results = tap_service.search(query)
...
DALQueryError: Maximum execution time (60 s) reached. Job aborted.
>>> tap_results = tap_service.search(query, maxrec=100)
DALOverflowWarning: Partial result set. Potential causes MAXREC, async storage space, etc.
>>> len(tap_results)
100
```
As for the TOP limited query:
```
>>> query1 = "select TOP 2500 * from v_public_observations"
>>> len(tap_service.search(query1))
2500
>>> len(tap_service.search(query1, maxrec=300))
DALOverflowWarning: Partial result set. Potential causes MAXREC, async storage space, etc.
300
>>> len(tap_service.search(query1, maxrec=3000))
2500
```
Contributor guide
Research direction
Start at astroquery/utils/tap/core.py line 272 and trace how launch_job applies maxrec to TAP queries, comparing the reported plain and TOP-limited examples with pyvo's TAPService behavior. Done means the hardwired 2000 behavior is removed or consistently configurable, while the documented maxrec examples behave as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- api, backend-api-design
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100