s4cmd 2.0.1 leaks memory for large uploads

Hi there:

We are using s4cmd to push, in parallel, about 70k files into an S3 bucket. The files vary in size from very small (10s of bytes) to somewhat large (11GB). On a recent run, s4cmd consumed nearly 80GB of memory before being caught by the Linux OOM killer.

To test, I created a smaller pool of 2,500 and 1,000 files and modified s4cmd to sleep before exiting so we could get an accurate RSS measurement.

The 2,500 file pool was 6.9GB on disk. The 1,000 file pool was 2.5GB on disk.

After successfully uploading both pools, RSS for the s4cmd process was roughly equal to the total size of files uploaded:

The 2.5GB pool ended up with an RSS of 2.7GB.

The 6.1GB pool ended up with an RSS of 7.3GB.

The size of the files being uploaded, not the total number of files, is the driving factor. Uploading 5,000 2 byte files only drives the RSS to .16GB (~167MB).

Specifying a very small max-singlepart-upload-size (for example, 1048576 bytes) does not change the memory usage.

After bisecting the upload() method in s4cmd.py, it appears the leak or cycle is happening inside of the boto3 put_object() call. Commenting out the put_object() keeps memory usage stable.

Importing and periodically calling gc.collect() also keeps memory usage in check.

I have not put together a test case just using boto3 to see if the leak occurs there. But I can't be the only person who would like to use s4cmd to upload several hundreds of GB of files.

Is there already a known workaround for this problem?

The environment is:

python 2.7.12
s4cmd 2.0.1
boto3 1.4.4
botocore 1.5.24

Start with the upload() method in s4cmd.py and reproduce the reported memory growth using the 1,000- and 2,500-file pools. Compare behavior around the boto3 put_object() call, including the reported effect of gc.collect(), across the stated Python, s4cmd, boto3, and botocore versions. Done means large uploads no longer retain memory roughly equal to the uploaded data.

Written by the indexing model from the issue text.