Should we increment minted S3 download urls expiry to avoid avoidable interaction with AWS?
- Dominant language
- Python
- Stars
- 26
- Forks
- 21
- Avg merge
- 4d 23h
- Merged PRs (30d)
- 15
Description
Instigated by
- https://github.com/dandi/dandi-archive/issues/1891
apparently we cache for up to 5 seconds and otherwise -- send a new one:
```shell
❯ pl=''; for i in {1..10}; do echo -n "$i $(date) "; l=$(curl --silent -I 'https://api.dandiarchive.org/api/dandisets/000891/versions/0.240215.0831/assets/d23e3ee7-8982-42dd-bd1d-fcbfd5290e04/download/' | grep Location); if [ "$l" != "$pl" ]; then echo -n "$l != $pl"; fi; pl=$l; echo; done
1 Thu Mar 21 11:58:37 AM EDT 2024 Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155837Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d71546e7de8 != 538a8efdfc45f8af07094a13fac0bb299df36e395c36a8090
2 Thu Mar 21 11:58:37 AM EDT 2024
3 Thu Mar 21 11:58:37 AM EDT 2024
4 Thu Mar 21 11:58:37 AM EDT 2024
5 Thu Mar 21 11:58:37 AM EDT 2024 Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155838Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e5b513d7ab4 != Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155837Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d71546e7de8d7d7538a8efdfc45f8af07094a13fac0bb299df36e395c36a8090
6 Thu Mar 21 11:58:38 AM EDT 2024
7 Thu Mar 21 11:58:38 AM EDT 2024
8 Thu Mar 21 11:58:38 AM EDT 2024
9 Thu Mar 21 11:58:38 AM EDT 2024
10 Thu Mar 21 11:58:38 AM EDT 2024 Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155839Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=63a77c1475 != Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155838Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e5b513d7ab4e605e57f84288a0078c0128898d22336fe2674e21c178be570d83
```
so in case of that repepetive bombardment for the same asset `/download/` for some reason - we would keep minting those urls quite often. Could/should we increase caching period given that they are to expire in 3600 seconds?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the API /download/ entry point described in the issue and trace how signed S3 URLs are cached and minted. Compare the cache duration with the 3600-second URL expiry, then determine and document the desired behavior for repeated requests; the payload names no source file or test to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, django, python
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100