aws / aws/aws-sdk-php

uploading large file leaves too much tmp files

Open
#3,210 3 comments 0 reactions 1 assignee Claimed by @stobrien89 View on GitHub
guidance
Dominant language
PHP
Stars
6.2k
Forks
1.3k
Avg merge
2h 53m
Merged PRs (30d)
7

Description

### Describe the bug

When uploading a large file (170GiB), using nextcloud and S3 backend, with MultiPart enabled, the aws code uses temp files to uploads. The issue is that it takes the same amount of files twice in disk, once to received the whole file, and once splitted as multipart split. This results in a requirement of 170GiB * 2 as temporary disk in my kubernetes cluster.

Isn't it possible to read and send as-you-go, in streaming mode, and cleaning the split part after having sent the multipart successfully, instead of waiting for a `fclose` of the input stream?

https://github.com/aws/aws-sdk-php/blob/3.360.1/src/S3/StreamWrapper.php#L32 says
```
* Because Amazon S3 requires a Content-Length header, write only streams will
* maintain a 'php://temp' stream to buffer data written to the stream until
* the stream is flushed (usually by closing the stream with fclose).
```

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

1/ The input file is read in a buffer in temp dir and after each read, the read data is discarded
2/ In a multipart upload, as soon as one split part is sent, the temporary file is deleted to free space.

### Current Behavior

1/ The input file of 170GiB is stored in `/tmp/phpXXXX`.
2/ In a multipart upload, all temporary files of split parts stays until the end of the whole upload.

### Reproduction Steps

Not an easy test case, to be improved:
1. deploy nextcloud
2. configure an external storage as s3 backend a cloud provider (scaleway, gcs, aws, etc.)
3. use rclone to send to nextcloud using webdav.

### Possible Solution

Put `unlink` carefully at the multiupload code somewhere?

### Additional Information/Context

_No response_

### SDK version used

3.349

### Environment details (Version of PHP (`php -v`)? OS name and version, etc.)

8.3.27

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.