Duke-GCB / Duke-GCB/DukeDSClient
Speed up uploading many files
- Dominant language
- Python
- Stars
- 5
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
A user reported uploading 3100 files took 2 hours.
A zipped version of these files took less than a minute.
**Current Implementation Profiling**
Profiling a smaller example of 12 folders and 17 files took **45 seconds**.
35 seconds was on the main process waiting on HTTP requests.
Creating the project took 1.2 seconds.
Each folder took average of 0.4 seconds.
Each file took average of 1.7 seconds(POST uploads: 0.6, PUT uploads_complete 0.75, files: 0.3s).
Creating the upload url and and sending content to swift store are handled on another process.
**Persistent Connection Profiling**
requests has support for using urllib3 persistent connections: http://docs.python-requests.org/en/master/user/advanced/
Profiling the same 12 folders and 17 files took **28 seconds**.
18 seconds are spent waiting on HTTP requests.
Creating the project took 0.1 seconds.
Each folder took average of 0.1 seconds.
Each file took average of 0.9 seconds(POST uploads: 0.3, PUT uploads_complete 0.5, files: 0.1s).
Even with this improvements I estimate it would still take an hour to upload 3100 files.
**Other Ideas**
Uploading multiple files at once.
Changes to DukeDS API for batch file creation.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.