aws / aws/aws-cli

Inefficient CPU usage of `aws s3 cp`

Open
#2,791 6 comments 3 reactions 0 assignees View on GitHub
feature-request p3 s3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

This issue arises primarily when copying a list of files from S3. My understanding is that the suggested approach is to copy the files individually by invoking `aws s3 cp`, for example:

cat files.txt | parallel aws s3 cp s3://remote/path/{/} /local/path/{/}

In our experience, for files under about 1MB (we work with large image datasets), copy time is _CPU bound_ by the `aws` process.

By comparison, the following script which serves as an alternative to `aws s3 cp` takes about _1/10th_ the CPU usage by my measurements (and thus downloads files much faster):

#!/bin/bash
contentType="text/html; charset=UTF-8"
date="`date -u +'%a, %d %b %Y %H:%M:%S GMT'`"
string="GET\n\n${contentType}\n\nx-amz-date:${date}\n${1}"
signature=`echo -en $string | openssl sha1 -hmac "${AWS_SECRET_KEY}" -binary | base64`
curl -o ${2} -s \
-H "x-amz-date: ${date}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${AWS_ACCESS_KEY}:${signature}" \
"https://s3.amazonaws.com${1}"

Where `${1}` is the S3 input path and `${2}` is the local output path.

I profiled the `aws s3 cp` command and it seems that most of the time is spent by the Python interpreter initializing the execution environment. If there can't be anything done to speed this up, it would be helpful to have an aws command to copy a list of files in parallel to avoid re-occurring this compute cost. This would appear possible as `aws s3 sync` doesn't consume nearly as much CPU, but it doesn't offer an interface suitable for copying a specific list of files.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.