Inefficient CPU usage of `aws s3 cp`
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
This issue arises primarily when copying a list of files from S3. My understanding is that the suggested approach is to copy the files individually by invoking `aws s3 cp`, for example:
cat files.txt | parallel aws s3 cp s3://remote/path/{/} /local/path/{/}
In our experience, for files under about 1MB (we work with large image datasets), copy time is _CPU bound_ by the `aws` process.
By comparison, the following script which serves as an alternative to `aws s3 cp` takes about _1/10th_ the CPU usage by my measurements (and thus downloads files much faster):
#!/bin/bash
contentType="text/html; charset=UTF-8"
date="`date -u +'%a, %d %b %Y %H:%M:%S GMT'`"
string="GET\n\n${contentType}\n\nx-amz-date:${date}\n${1}"
signature=`echo -en $string | openssl sha1 -hmac "${AWS_SECRET_KEY}" -binary | base64`
curl -o ${2} -s \
-H "x-amz-date: ${date}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${AWS_ACCESS_KEY}:${signature}" \
"https://s3.amazonaws.com${1}"
Where `${1}` is the S3 input path and `${2}` is the local output path.
I profiled the `aws s3 cp` command and it seems that most of the time is spent by the Python interpreter initializing the execution environment. If there can't be anything done to speed this up, it would be helpful to have an aws command to copy a list of files in parallel to avoid re-occurring this compute cost. This would appear possible as `aws s3 sync` doesn't consume nearly as much CPU, but it doesn't offer an interface suitable for copying a specific list of files.
Contributor guide
Assessment
This issue has not been assessed yet.