aws / aws/aws-cli

S3 sync output log individual file size

Open
#7,536 2 comments 0 reactions 0 assignees View on GitHub
feature-request p3 s3sync
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the feature

Currently the AWS CLI S3 sync command using option --no-progress outputs log lines as follows e.g.
upload: src_object_path_1 to dst_object_path_1
upload: src_object_path_2 to dst_object_path_2

I would like to propose the AWS S3 sync command also outputs the size of each transferred file in the output log lines as follows e.g.
upload: src_object_path_1 to dst_object_path_1 (41234 bytes)
upload: src_object_path_2 to dst_object_path_2 (798 bytes)

Unfortunately the data transfer file size provided by the progress bar does not offer enough fidelity for my needs.

### Use Case

I currently capture all my AWS CLI S3 sync command output to log files and from these I generate reports on the file names, number of files and the times the files were transferred, with this extra output of file size I will be also be able to report on much more granular volumetric data.

I have investigated using AWS S3 Storage Lens and AWS S3 Request Metrics but these do not quite provide what I am after.

### Proposed Solution

I have looked into the AWS CLI source code and installed the build environment where I have been playing around with this file without much success.
https://github.com/aws/aws-cli/blob/2.9.8/awscli/customizations/s3/results.py

The information I want to present is available from a variable called `result.total_transfer_size` in function `_record_queued_result()`, however I can't seem to make that variable content available to a modified version of the function called `_get_transfer_location()` and its dependencies.

I would be more confident if someone with better Python skills than me would look into this.

### Other Information

I have not considered creating an extra command line option for this new feature and I don't know if enabling this new feature by default will upset anyone.

However, the counter argument is that AWS CLI S3 sync command in its current form does not honor the --output json option or any other output filtering capability so it is already behaving a little different to other AWS commands. With my proposal being an additional field appended to the end of the existing log line output then this may have less impact to those not expecting to see this new field, but this is an assumption.

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CLI version used

2.9.8

### Environment details (OS name and version, etc.)

CentOS-8 Stream

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.