astropy / astropy/astroquery

MAST download should efficiently fallback on existing downloads

Open
#2,820 6 comments 0 reactions 0 assignees View on GitHub
bug mast
Dominant language
Python
Stars
791
Forks
451
Avg merge
1d 3h
Merged PRs (30d)
4

Description

Example use case:

```python
from astroquery.mast import Observations

files = ['jw02727-o002_t062_nircam_clear-f090w_i2d.fits',
'jw02727-o002_t062_nircam_clear-f277w_i2d.fits'
]

for fn in files:
uri = f"mast:JWST/product/{fn}"
result = Observations.download_file(uri, local_path=fn)
```

I already have cached copies of these files.

Problem: When MAST server performance is degraded, it takes a really long time for `download_file` to complete when I just really want it to grab the cached copy. Even when successful, it would end up just grabbing the cached copy anyway.

Workaround: I have to avoid using `download_file` altogether, hunt down where that file was downloaded to, and use it directly.

Desired solution: When MAST server takes too long to respond (maybe user can set this timeout?), automatically stops and grab a cached copy. Throw a warning if you like. Throw error as before if no cached copy found.

cc @bmorris3

Related:

* https://github.com/lightkurve/lightkurve/issues/1159

Contributor guide

Open the contributing guide

Research direction

Start at the Observations.download_file entry point and trace how MAST URIs are fetched and how existing downloads are located. Determine how a configurable timeout should trigger cached-file use, while preserving the existing error when no cached copy exists; verify the behavior with the two JWST filenames from the example under degraded-server conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.