Add a MAST Observation method that gets the list of URLs for a table product but doesn't actually download them
- Dominant language
- Python
- Stars
- 791
- Forks
- 451
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
I've run into some use cases recently where the `download_products`/`download_file` methods in `mast.Observation` aren't meeting my needs. Specifically there are two use cases I'm thinking about:
1. Where I want to download the data products as a batch but want to put them in a layout that doesn't match `download_products`. I can do this by looping over the product table and using `download_file`, but it ends up rather awkward because `download_file` expects a URI and sometimes the table references start getting complex
2. Sometimes I don't want to download the data at all, but instead want to download using `requests` or similar straight into memory (especially useful for association files)
3. If I am writing something to run in AWS there are some shortcuts that are good to consider if it turns out some of the products are in S3.
I think all of those can be satisfied by a function that's like `download_products`, but instead of actually doing the download, provides a list of URLs for the products (which might be s3 if the cloud functionality is turned on, or might be regular MAST urls). This is basically what the existing `curl_flag` option in `download_products` does, but instead of downloading a curl script, would return the URLs the curl script would result in.
IMHO, this should be a separate method not another option to `download_products`/`download_file`, since it doesn't actually do any downloading, but I imagine it could re-use most of the functionality that is in those methods to not have to implement much code.
cc @jaymedina who I had some out-of-band conversation with on this already
Contributor guide
Research direction
Start with mast.Observation.download_products and download_file, especially the existing curl_flag behavior. Trace how product-table references become regular MAST or S3 URLs, then define the new method's output and verify that it returns the URLs without downloading; completion should cover the batch use case and cloud-enabled behavior described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100