intermine / intermine/intermine_compose
Support resumable/chunked uploads
- Dominant language
- Python
- Stars
- 4
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
We considered supporting uploading via the CLI intermine_boot client, but after some discussion found that it would be more user-friendly and probably just as reliable to use chunked uploads in the browser.
One good library that provides this is [flow.js](https://github.com/flowjs/flow.js).
We will need to have the upload endpoint support the [API](https://github.com/flowjs/flow.js#how-do-i-set-it-up-with-my-server).
The next question would be where to save the files.
- Upload to intermine_compose and save file to local filesystem **(how it's done currently)**
- Upload to intermine_compose and use [GCP’s app engine cloud client library](https://cloud.google.com/appengine/docs/standard/python3/using-cloud-storage) to save to cloud bucket **(we should probably move over to this, as the files will be large and numerous; the previous approach should still be used when this is not available due to running locally with intermine_boot)**
- Upload directly to Google Cloud Storage bucket **(a different alternative; might be more difficult and not sure if it's worth the extra effort)**
- This can be done for large files using either flow.js ([issue](https://github.com/flowjs/flow.js/issues/170), [PR](https://github.com/flowjs/flow.js/pull/276)) or GCS directly via resumable uploads API ([info](https://cloud.google.com/storage/docs/resumable-uploads), [guide](https://cloud.google.com/storage/docs/performing-resumable-uploads))
Note that the files need to be available to both the wizard and InterMine instance without being duplicated. (With a GCP bucket: after [acquiring an OAuth token](https://cloud.google.com/storage/docs/authentication), files can be [downloaded from buckets with a GET request](https://cloud.google.com/storage/docs/downloading-objects#rest-download-object).) This could be a problem with our current Auth approach, so we may need to explore using Daniela’s OAuth server
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.