astropy / astropy/astroquery

Add a MAST Observation method that gets the list of URLs for a table product but doesn't actually download them

Open
#2,850 1 comment 1 reaction 0 assignees View on GitHub
mast
Dominant language
Python
Stars
791
Forks
451
Avg merge
1d 3h
Merged PRs (30d)
4

Description

I've run into some use cases recently where the `download_products`/`download_file` methods in `mast.Observation` aren't meeting my needs. Specifically there are two use cases I'm thinking about:

1. Where I want to download the data products as a batch but want to put them in a layout that doesn't match `download_products`. I can do this by looping over the product table and using `download_file`, but it ends up rather awkward because `download_file` expects a URI and sometimes the table references start getting complex
2. Sometimes I don't want to download the data at all, but instead want to download using `requests` or similar straight into memory (especially useful for association files)
3. If I am writing something to run in AWS there are some shortcuts that are good to consider if it turns out some of the products are in S3.

I think all of those can be satisfied by a function that's like `download_products`, but instead of actually doing the download, provides a list of URLs for the products (which might be s3 if the cloud functionality is turned on, or might be regular MAST urls). This is basically what the existing `curl_flag` option in `download_products` does, but instead of downloading a curl script, would return the URLs the curl script would result in.

IMHO, this should be a separate method not another option to `download_products`/`download_file`, since it doesn't actually do any downloading, but I imagine it could re-use most of the functionality that is in those methods to not have to implement much code.

cc @jaymedina who I had some out-of-band conversation with on this already

Contributor guide

Open the contributing guide

Research direction

Start with mast.Observation.download_products and download_file, especially the existing curl_flag behavior. Trace how product-table references become regular MAST or S3 URLs, then define the new method's output and verify that it returns the URLs without downloading; completion should cover the batch use case and cloud-enabled behavior described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.