"Same filesystem" option to avoid crossing filesystem boundaries
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
I'm trying to sync a subset of files from an S3 bucket to `/` with a command like:
`aws s3 sync s3://bucketname/machine/foo/root_fs /`
in that `root_fs` folder/prefix is a tree of files to copy/update on the destination, e.g. `var/opt/foo/etc/bar` which should end up at `/var/opt/foo/etc/bar`, `opt/foo/etc/baz` which should end up at `/opt/foo/etc/baz`.
However, the file list generation appears to be going into procfs (under /proc) etc, and getting itself all confused and stuck, e.g.:
```
2020-06-08 10:37:57,816 - MainThread - awscli.customizations.s3.filters - DEBUG - //proc/1/cwd/proc/1/cwd/proc/1/cwd/proc/1/cwd/proc/1/cwd/proc/1/cwd/proc/1/cwd/proc/1/cwd/proc/1/cwd/.cpanm/work/1590503255.22989/Image-ExifTool-11.70/html/TagNames/PostScript.html did not match include filter: /var/opt/foo
```
(As you can see by the mention of include filters, I did try adding e.g. `--exclude='*' --include=/var/opt/foo/*` to the command to see if it would get the file generation step to exclude stuff that I hadn't explicitly mentioned, but it seems that comes later.)
**Describe the solution you'd like**
An option similar to rsync's `--one-file-system` option ("don't cross filesystem boundaries") so that s3 sync will stay on the filesystem the destination path is on, and not cross over to other filesystems which may be mounted below it (e.g. procfs at /proc).
Even nicer would be for running sync in this way without `--delete` options to not have to assemble a list of files in the destination *at all*, and instead just walk the list of files from the source and sync them to the appropriate place - but since using `--delete` to delete files that exist at the destination but not the source (which I'm not doing for obvious reasons!) requires knowing about such files, I can see why it does so.
**Describe alternatives you've considered**
I've tried using `--exclude` and `--include` options to have the file list generation only consider the actual paths in question, but that doesn't seem to have helped, it's still getting itself in that loop where it's descending under the procfs filesystem and looping (because `proc/1/cwd` is a symlink to `/`).
I should be able to work around it by looping through each prefix in the S3 bucket which I want to sync, and calling s3 sync individually for each with the target path set, rather than trying to do it the "easy way".
Contributor guide
Assessment
This issue has not been assessed yet.