developmentseed / developmentseed/sat-ml-training

Getting big files into Colab without Download

Open
#22 0 comments 0 reactions 1 assignee Claimed by @wildintellect View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
27
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Was working with the Zindi data, it's not really feasible for participants to download the data themselves and then upload to their google drives, could easily take hours. As a workaround we are putting all the data unzipped on Google Shared Drive accessible to participants. This is a common problem in general for ML work.

However it would be really awesome to use Colab to download the data directly to the cloud and unzip.
* It takes authentication to the Zindi platform to download, so using requests library is a little more complicated
* The links in the page are obsfucated, "Note: If you want to download the satellite data using a script, you can contact us for a permanent URL"

Alternatives:
* Get the Scenes from GEE or AWS Public Bucket
* Have the data ready to use (plan to do so anyways)
* There's some interesting Chrome plugins to save to drive directly, not sure which are safe to use.

This is a wishlist item if we've got all the other important stuff done already, but important to talk about.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.