aws / aws/aws-sdk-cpp

Huge RAM usage on big file uploads

Open
#2,856 7 comments 0 reactions 0 assignees View on GitHub
feature-request p2
Dominant language
C++
Stars
2.2k
Forks
1.2k
Avg merge
3d 14h
Merged PRs (30d)
12

Description

### Describe the bug

I want to uploada big file. So I wanted to up the partSize of my s3crtclient Configuration.
But, it seems the RAM consumption of my process is a direct multiple (around 20x) of that value. So when I tried 50MB, my process was taking 1GB of RAM.

### Expected Behavior

Uploading files should be simple enough that it consumes less RAM.

### Current Behavior

It uses 20x the RAM of the part size. For a huge upload that is too much. And that means I cannot do more than 1 in parallel.

### Reproduction Steps

see description

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS CPP SDK version used

1.11.258

### Compiler and Version used

Apple clang version 15.0.0 (clang-1500.1.0.2.5)

### Operating System and version

macOS Sonoma 14.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.