aws / aws/aws-cli

s3 sync: files with excluded prefixes checked client side

Open
#7,653 9 comments 0 reactions 0 assignees View on GitHub
p3 s3sync
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

We sync files from locally to the root of an S3 bucket. There is an existing folder in that same bucket, which we `--exclude` for this specific sync - that folder has over 1.800.000 objects.

The sync takes about 16 minutes - while doing nothing other than listing all objects including those from the mentioned excluded folder and checking against the exclude rules.

### Expected Behavior

The `--exclude` option should lead to the excluded paths to be excluded during the call to list the S3 objects.

This, way the exclude check would not need to be performed client-side, but rather server-side.

### Current Behavior

The sync lists ALL objects in the entire bucket to then decide on the excludes client-side.

### Reproduction Steps

1. Create bucket `test-bucket` (choose distinct name)
2. Add folder `test` with many random files (e.g. millions)
3. Run sync to the root of that folder, excluding the `test` folder
```
aws s3 sync --delete --debug . s3://test-bucket --exclude "test/*"
```

### Possible Solution

The S3 [`ListObjectsV2`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) endpoint that is used does not seem to expose filter options.

### Additional Information/Context

_No response_

### CLI version used

2.9.19

### Environment details (OS name and version, etc.)

Ubuntu 22.04.1 (on GitHub Actions)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.