astropy / astropy/pyvo

Datalink getdataset too eager

Open
#554 2 comments 0 reactions 0 assignees View on GitHub
component: SIA
Dominant language
Python
Stars
88
Forks
64
Avg merge
38m
Merged PRs (30d)
1

Description

Consider this script:

```
import pyvo

ACCESS_URL = "http://dc.g-vo.org/lswscans/res/positions/siap/siap.xml?"
svc = pyvo.sia.SIAService(ACCESS_URL)
images = svc.search((11,35), size=(0.1, 0.1))
images[0].cachedataset()
```

Warning: This will download at least 100 Megabytes with current pyVO.

It shouldn't, though.

This is one of the SIA1 services that return cutouts as access_urls. However, there is a datalink service on the result, too. That service only uses the full data ids, not any cutout instructions. That is, #this can be a FITS of 1 GB or so, whereas the accref can point to just a few epsilons.

Now, you can argue that if I can't make #this exactly the dataset that is described in the SIA row then I shouldn't make the datalink in the first place; but I claim this arrangement is useful in practice (e.g., people can adjust their cutouts, or pull the full image if that's what they want) and at least not *totally* wrong.

On the other hand, pyVO's current behaviour, that is, to prefer datalink's #this over the access reference given in the SIAP response, is weird (to say the least). I'm not even sure why we're doing it. It seems the original code was written by @funbaker, who I think has dropped out of pyVO development, so I don't think we can ask him.

Anyway, I'd suggest to prefer SIAP's access reference and only fall back to datalink if that's not going anywhere. Technically, I'd simply turn around the logic in adhoc.DatalinkRecordMixin.getdataset: try super() first and only if that exceptions out try datalink's #this.

Does anyone feel that would be grossly wrong?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with adhoc.DatalinkRecordMixin.getdataset and inspect the superclass path used for the SIAP access reference. Reproduce the provided SIAService example, then verify that the SIAP access reference is preferred and datalink's #this is used only when the former fails, without eagerly downloading the larger dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.