Upload is slower for very large because of md5 calculation
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 209
- PR merge metrics
- No merged PRs in 30d
Description
For larger file uploads, takes too long mainly because of file_hash (md5) calculation.
This is line is the line responsible for this
mpu = bucket.initiate_multipart_upload(s3url.path, metadata = {'md5': self.file_hash(source), 'privilege': self.get_file_privilege(source)})
https://github.com/bloomreach/s4cmd/blob/master/s4cmd.py#L1043
I've read in aws documentation that this md5 metadata is not necessary, I suspect this md5 calculation exists mainly because local md5 of file is different from the generated by S3.
It is used in `sync_check`:
('md5' in remoteKey.metadata and remoteKey.metadata['md5'] == localmd5)
It is useful mostly for calls that use sync option.
But sometimes, like in my case, the user of s4cmd may not care about sync, since he may using the -f (force) option.
I thought about two possible solutions:
- One argument to just disable this file_hash execution like "--disable-multipart-meta-md5"
- The second one is more ambitious, and I don't know if it is really possible:
Don't calculate the md5, but uses s3's generated md5 for calculation. (We will need to reproduce s3's md5 calculation)
S3 calculates the s3 etag for multipart uploads using a md5 of the md5 of splitted parts. The algorithm is described in these links:
http://stackoverflow.com/questions/6591047/etag-definition-changed-in-amazon-s3
http://permalink.gmane.org/gmane.comp.file-systems.s3.s3tools/583
http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb
https://forums.aws.amazon.com/thread.jspa?messageID=203510
The thing that is missing from to recalculate s3's md5 for multipart is the SPLIT_SIZE.
But s3's e-tag for file has a sufix -NUMBER_OF_PARTS.
```
s3cmd --recursive --list-md5 ls s3://YOURBUCKET/YOURDIR/
2015-02-13 05:02 9791602688 f938b15b2edef7d2c23542814bdcb6af-187 s3://FILEPATH
```
I guess with this info size (and number of parts), we could use to calculate like amazon's s3 does
In the example above:
```
parts = 187
file_size = toMB(9791602688.0D) # 9338.0
SPLIT_SIZE = ((file_size - (file_size % parts)) / parts) + 1
SPLIT_SIZE = 50 # MB
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Read s4cmd.py around line 1043, including file_hash and the sync_check use of the md5 metadata, then review the multipart-upload behavior described in the issue. Done should be a clearly bounded approach for avoiding unnecessary hashing while preserving sync behavior, verified against large multipart uploads and sync checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100