aws / aws/aws-cli

'aws s3 sync' confused by multiple successive slashes in S3 bucket.

Open
#6,253 5 comments 0 reactions 0 assignees View on GitHub
bug p2 s3sync
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

- [X] I've gone though the [User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) and the [API reference](https://docs.aws.amazon.com/cli/latest/reference/)
- [X] I've searched for [previous similar issues](https://github.com/aws/aws-cli/issues) and didn't find any solution

**Describe the bug**
`aws s3 sync` can be confused by a valid bucket structure that is not possible to map into a local file system: `s3://mybucket/A` and `s3://mybucket//A` are distinct S3 objects, but are synced to the same local file, since path `./A` and `.//A` coincide.

**SDK version number**

```
aws-cli/2.1.28 Python/3.8.8 Linux/4.14.225-168.357.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off
```

**Platform/OS/Hardware/Device**
CloudShell/Posix

**To Reproduce (observed behavior)**
```
$ aws s3 mb s3://xx20210629
make_bucket: xx20210629
$ echo 1 > A
$ echo 1000 > B
$ aws s3 cp A s3://xx20210629/a
upload: ./A to s3://xx20210629/a
$ aws s3 cp B s3://xx20210629//a
upload: ./B to s3://xx20210629//a
$ aws s3 ls --recursive s3://xx20210629
2021-06-29 15:39:22 5 /a
2021-06-29 15:39:14 2 a
$ aws s3 sync s3://xx20210629 local
download: s3://xx20210629//a to local/a
download: s3://xx20210629/a to local/a
$ cat local/a
1
```

**Expected behavior**
It is a contrieved situation, but overwriting local files should avoided at all costs. BTW I'm reporting this because it took me a lot of time to understand what was going on. Please note also that `sync` is not guaranteed to be idempotent:
```
$ aws s3 sync s3://xx20210629 local
download: s3://xx20210629//a to local/a
$ cat local/a
1000
$ aws s3 sync s3://xx20210629 local
download: s3://xx20210629/a to local/a
download: s3://xx20210629//a to local/a
$ cat local/a
1000
```

So I would suggest
- error out or mangle file-name if multiple successive slashes are present in the S3 bucket.
- ensure that `aws s3 sync` is idempotent if called multiple times.

**Logs/output**
Not relevant

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.