aws / aws/aws-cli

aws s3 sync status takes time to calculate total/remaining files

Open
#7,957 3 comments 4 reactions 0 assignees View on GitHub
feature-request p3 s3 s3sync
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

Running `aws s3 sync [s3 folder with 8 files] [local folder] `
and for approx half of the runtime the status doesn't show real numbers of total/remaining files

### Expected Behavior

Status shown at the start:
Completed xxx GiB/~yyy GiB (zzz MiB/s) with **~8** file(s) remaining **(8 files)**

+ it will be nice to have the ETA shown as `Estimated time remaining: [dd] days [hh] hours [mi] minutes [ss] seconds`

### Current Behavior

While it copies the first 5 files in parallel the status shown is:
Completed xxx GiB/~yyy GiB (zzz MiB/s) with **~5** file(s) remaining **(calculating...)**

### Reproduction Steps

Upload 10 big files (10 GB each) to an s3 folder and run from and ec2 VM:
`aws s3 sync [s3 folder] [local folder]`

### Possible Solution

`aws s3 sync` should try first scanning for a few seconds all files/subfolders and calculate total number of files and size to quickly display it in the command status.

If scanning takes more than a few seconds then it should start syncing the files/subfolders and display `with total (calculating...) file(s) remaining (calculating...)` and eventually display the real total/remaining number after if finished scanning the entire folder tree.

### Additional Information/Context

_No response_

### CLI version used

2.11.25

### Environment details (OS name and version, etc.)

Windows Server 2022 Datacenter / Powershell

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.