aws / aws/aws-cli

Add rsync-like --update to s3 sync.

Open
#404 14 comments 13 reactions 0 assignees View on GitHub
feature-request p3 s3 s3sync s3syncstrategy
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

It would be very powerful if sync copied the behavior of [rsync](http://www.samba.org/ftp/rsync/rsync.html)'s --update flag. This would allows predictable syncing from multiple directories without clobbering newer files. As it behaves right now, sync will update a file if the size OR timestamp differs.

In this example I'm creating a two folders, each with one file (test_file) having different sizes. I then create a bucket, sync the older, then the newer, then the older. As you can see from the output, the newer file is overwritten.

```
mkdir srcA srcB;
echo "older" > srcA/test_file;
sleep 2;
echo "this is newer" > srcB/test_file;

alias aws='aws --region=us-east-1';
ACL="--acl public-read";
bucket=$(uuidgen);

aws s3api create-bucket $ACL --bucket $bucket
cd srcA; aws s3 sync $ACL . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file
cd srcB; aws s3 sync $ACL . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file
cd srcA; aws s3 sync $ACL . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file
aws s3 rm s3://$bucket/test/test_file
aws s3api delete-bucket --bucket $bucket
```

```
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> older
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> this is newer
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> older
```

With the --update flag in place like this:

```
cd srcA; aws s3 sync $ACL --update . s3://$bucket/test/; cd ..
cd srcB; aws s3 sync $ACL --update . s3://$bucket/test/; cd ..
cd srcA; aws s3 sync $ACL --update . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file
```

I would expect the following output:

```
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
# no output from the third command
=> this is newer
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.