intermine / intermine/intermine_compose

Support resumable/chunked uploads

Open
#26 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4
Forks
5
PR merge metrics
No merged PRs in 30d

Description

We considered supporting uploading via the CLI intermine_boot client, but after some discussion found that it would be more user-friendly and probably just as reliable to use chunked uploads in the browser.

One good library that provides this is [flow.js](https://github.com/flowjs/flow.js).
We will need to have the upload endpoint support the [API](https://github.com/flowjs/flow.js#how-do-i-set-it-up-with-my-server).

The next question would be where to save the files.
- Upload to intermine_compose and save file to local filesystem **(how it's done currently)**
- Upload to intermine_compose and use [GCP’s app engine cloud client library](https://cloud.google.com/appengine/docs/standard/python3/using-cloud-storage) to save to cloud bucket **(we should probably move over to this, as the files will be large and numerous; the previous approach should still be used when this is not available due to running locally with intermine_boot)**
- Upload directly to Google Cloud Storage bucket **(a different alternative; might be more difficult and not sure if it's worth the extra effort)**
- This can be done for large files using either flow.js ([issue](https://github.com/flowjs/flow.js/issues/170), [PR](https://github.com/flowjs/flow.js/pull/276)) or GCS directly via resumable uploads API ([info](https://cloud.google.com/storage/docs/resumable-uploads), [guide](https://cloud.google.com/storage/docs/performing-resumable-uploads))

Note that the files need to be available to both the wizard and InterMine instance without being duplicated. (With a GCP bucket: after [acquiring an OAuth token](https://cloud.google.com/storage/docs/authentication), files can be [downloaded from buckets with a GET request](https://cloud.google.com/storage/docs/downloading-objects#rest-download-object).) This could be a problem with our current Auth approach, so we may need to explore using Daniela’s OAuth server

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.