higlass / higlass/higlass-docker

Choose s3-fs-thing and implement

Open
#19 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
34
Forks
16
PR merge metrics
No merged PRs in 30d

Description

We'll need to store more data than fits comfortably on EBS. Options:
- EFS: but it's [expensive](https://aws.amazon.com/efs/pricing/): $0.30/GB-month. In their defense, it looks like they mirror between availability zones, and for some users that might be a big selling point.
- S3: [cheaper](https://aws.amazon.com/s3/pricing/): $0.023/GB-month
- https://github.com/s3fs-fuse/s3fs-fuse: but if you need less from your FS, points to:
- https://github.com/kahing/goofys: Check benchmarks: time to first byte is an order of magnitude smaller, and I think that's the critical measure for us.
- [Google Cloud Storage](https://cloud.google.com/storage/pricing): $0.02/GB-month: A bit cheaper, but I think we're comfortable with vendor lock-in?
- [Azure](https://azure.microsoft.com/en-us/pricing/details/storage/blobs/) in the same neighborhood.

Both [AWS](https://aws.amazon.com/s3/faqs/#sia_anchor) and [Google](https://cloud.google.com/storage/docs/storage-classes#nearline) offer a secondary tier for objects accessed less than once a month, and the cost is about half. Probably not worth the hassle, but it's there.

**Implementation**
- in the container? [possible](https://github.com/kahing/goofys/issues/118#issuecomment-262743941)... why not in production?
- in s3? Simpler, but configuration needs to be redone each time.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.