aws s3 ls allows listing by file prefix, but aws s3 sync does not
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
Hi,
I have an S3 Bucket that has a range of files like this:
```
0000-abcdef-ghiklm-nopqrs
1000-abcdef-ghiklm-nopqrs
2000-abcdef-ghiklm-nopqrs
3000-abcdef-ghiklm-nopqrs
a000-abcdef-ghiklm-nopqrs
b000-abcdef-ghiklm-nopqrs
c000-abcdef-ghiklm-nopqrs
d000-abcdef-ghiklm-nopqrs
```
Each prefix has millions of files in the bucket.
If I want to list files by a prefix, I can do this really efficiently, like so:
* `aws s3 ls s3://my-bucket/1` - Will only list files beginning with `1`
* `aws s3 ls s3://my-bucket/2` - Will only list files beginning with `2`
This is efficient, as it doesn't return any files without the appropriate prefix. The API handles the list, only returning files that start with `1` or `2`
---
The same isn't possible for the `aws s3 sync` command.
`aws s3 sync s3://my-bucket/1* s3://my-new-bucket/` or `aws s3 sync s3://my-bucket/1 s3://my-new-bucket/` does not work.
The only way to do it is through adding a `--exclude` `--include` parameter for example `aws s3 sync s3://my-bucket/ s3://my-new-bucket/ --exclude "*" --include "1*"`
But this is really inefficient, as it lists the whole contents of the bucket, and filters each item to determine whether it should match **client side**. This method makes perfect sense for normal include exclude operations like `--exclude "*" --include "*.txt"` but for prefixed files it is inefficient.
I know that folder/path prefixes should be used, but that isn't always possible. It is clear that the `aws s3 ls` command has the potential to return only files with a specific prefix, so could this behaviour be added into the sync command? Otherwise syncs between buckets take weeks instead of hours.
Contributor guide
Assessment
This issue has not been assessed yet.