Duke-GCB / Duke-GCB/DukeDSClient

Speed up uploading many files

Open
#80 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5
Forks
6
PR merge metrics
No merged PRs in 30d

Description

A user reported uploading 3100 files took 2 hours.
A zipped version of these files took less than a minute.

**Current Implementation Profiling**
Profiling a smaller example of 12 folders and 17 files took **45 seconds**.
35 seconds was on the main process waiting on HTTP requests.
Creating the project took 1.2 seconds.
Each folder took average of 0.4 seconds.
Each file took average of 1.7 seconds(POST uploads: 0.6, PUT uploads_complete 0.75, files: 0.3s).
Creating the upload url and and sending content to swift store are handled on another process.

**Persistent Connection Profiling**
requests has support for using urllib3 persistent connections: http://docs.python-requests.org/en/master/user/advanced/
Profiling the same 12 folders and 17 files took **28 seconds**.
18 seconds are spent waiting on HTTP requests.
Creating the project took 0.1 seconds.
Each folder took average of 0.1 seconds.
Each file took average of 0.9 seconds(POST uploads: 0.3, PUT uploads_complete 0.5, files: 0.1s).
Even with this improvements I estimate it would still take an hour to upload 3100 files.

**Other Ideas**
Uploading multiple files at once.
Changes to DukeDS API for batch file creation.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.