computablelabs / computablelabs/ffa-datatrust

Determine file storage strategy in API

Open
#2 5 comments 0 reactions 1 assignee Claimed by @rekibnikufesin View on GitHub
enhancement help wanted
Dominant language
Python
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

We need to determine how uploaded files are stored by the API.
Some initial thoughts, input welcome:
- API receives the file, submits the hash to protocol
- pulls any metadata/mime-types from the file
- creates a thumbnail image --> sends to public S3 bucket for discoverability
- in dynamodb: store mime-type, S3 URL, hash, thumbnail path, file size (bytes), metadata provided by supply user
- should we store this in S3? Or should it be a local file system to the API...?
- S3 Pros:
- S3 is cheap and can easily be used as a CDN source
- delivery payload can be a signed URL with expiration
- S3 Cons:
- S3 is slow as a file system
- File system pros:
- fast 🐇
- File system cons:
- syncing with CDN can be a pain
- need multiple file systems or shared file system for supporting highly available API instances
- this rules out ECS Fargate. We'd either build on EC2 or EC2 based ECS

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.