bcgov / bcgov/nr-rfc-processing
Speedup Object Storage Uploads (Async/Multiprocess)
Open
enhancement
- Dominant language
- Python
- Stars
- 4
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Currently the upload to object storage works on a single thread. Could significantly reduce upload time by making the upload process async.
See example for how to do that here:
download_granules.py which uses the multiprocess module. Similar approach could be used for syncing to object storage.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.