dandi / dandi/dandi-archive

Should we increment minted S3 download urls expiry to avoid avoidable interaction with AWS?

Open
#1,901 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
26
Forks
21
Avg merge
4d 23h
Merged PRs (30d)
15

Description

Instigated by
- https://github.com/dandi/dandi-archive/issues/1891

apparently we cache for up to 5 seconds and otherwise -- send a new one:

```shell
❯ pl=''; for i in {1..10}; do echo -n "$i $(date) "; l=$(curl --silent -I 'https://api.dandiarchive.org/api/dandisets/000891/versions/0.240215.0831/assets/d23e3ee7-8982-42dd-bd1d-fcbfd5290e04/download/' | grep Location); if [ "$l" != "$pl" ]; then echo -n "$l != $pl"; fi; pl=$l; echo; done
1 Thu Mar 21 11:58:37 AM EDT 2024 Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155837Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d71546e7de8 != 538a8efdfc45f8af07094a13fac0bb299df36e395c36a8090
2 Thu Mar 21 11:58:37 AM EDT 2024
3 Thu Mar 21 11:58:37 AM EDT 2024
4 Thu Mar 21 11:58:37 AM EDT 2024
5 Thu Mar 21 11:58:37 AM EDT 2024 Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155838Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e5b513d7ab4 != Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155837Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d71546e7de8d7d7538a8efdfc45f8af07094a13fac0bb299df36e395c36a8090
6 Thu Mar 21 11:58:38 AM EDT 2024
7 Thu Mar 21 11:58:38 AM EDT 2024
8 Thu Mar 21 11:58:38 AM EDT 2024
9 Thu Mar 21 11:58:38 AM EDT 2024
10 Thu Mar 21 11:58:38 AM EDT 2024 Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155839Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=63a77c1475 != Location: https://dandiarchive.s3.amazonaws.com/blobs/775/0c1/7750c16b-025f-4f8c-be3e-e67bee819dc4?response-content-disposition=attachment%3B%20filename%3D%22sub-FB2355_ses-FB2355_image.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240321%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240321T155838Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e5b513d7ab4e605e57f84288a0078c0128898d22336fe2674e21c178be570d83
```

so in case of that repepetive bombardment for the same asset `/download/` for some reason - we would keep minting those urls quite often. Could/should we increase caching period given that they are to expire in 3600 seconds?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the API /download/ entry point described in the issue and trace how signed S3 URLs are cached and minted. Compare the cache duration with the 3600-second URL expiry, then determine and document the desired behavior for repeated requests; the payload names no source file or test to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, django, python
Domain
api, backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.