astropy / astropy/astroquery

CADC Grabbing cutouts from a get_image_list URL

Open
#1,886 9 comments 0 reactions 0 assignees View on GitHub
cadc
Dominant language
Python
Stars
791
Forks
451
Avg merge
1d 3h
Merged PRs (30d)
4

Description

Hi, I've written a small bit of code to query the CADC archive for HST data cutout. I use get_image_list to retrieve the image filenames, but I've noticed that the code I came up with to grab the fits data cutout isn't always reliable for all queries. Sometimes I get back an empty list. I've traced this to a HTTP 409 error that I seem to be getting. Any clues on what might be going wrong here?

A minimal example:
```
import requests
def get_images_from_list(image_list, show_progress=False):
"""Returns lists of HDUs from given image list URLS"""
from astroquery.utils import commons
fits_list = []
filenames = [commons.FileContainer(url, encoding='binary',show_progress=show_progress) for url in image_list]
for fn in filenames:
try:
fits_list.append(fn.get_fits())
except requests.exceptions.HTTPError as err:
# Catch HTTPError if user is unauthorized to access file
print("{} - Problem retrieving the file: {}".format(str(err), str(err.url)))
pass
return fits_list

#This URL fails
image_list_fail = ['https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=mast%3AHST%2Fproduct%2Fj8xi6ajdq_flc.fits&RUNID=p8dbfo12deijxaag&POS=CIRCLE+150.298095703125+2.7121429443359375+0.0015277777777777779']

#This URL doesn't
image_list_pass = ['https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=mast%3AHST%2Fproduct%2Fjcor23010_drz.fits&RUNID=chsl0cmf1fwzq90d&POS=CIRCLE+150.33006909892683+2.07031305099778+0.0015277777777777779']

#Grab fits images
fits_list = get_images_from_list(image_list_pass, show_progress=False)
print("Success!")
fits_list = get_images_from_list(image_list_fail, show_progress=False)

```

Contributor guide

Open the contributing guide

Research direction

Start with the minimal Python example and compare the failing and passing CADC sync URLs, focusing on the HTTP 409 response and the use of get_image_list and commons.FileContainer.get_fits(). Reproduce both requests and determine what change would make retrieval reliable for the failing query; done means the cutout is returned instead of an empty list.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.