bloomreach / bloomreach/s4cmd

get or put or sync files by nested wildcards

Open
#148 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

Say I have following directory structure on my S3 bucket:

```
/
dir1/
subdir/

main.conf
test.conf
dir2/
subdir/

main.conf
test.conf
dir3/
subdir/

main.conf
test.conf
```

Now I want to download only those `*.conf` files, edit them and upload back without interfering with `` in denoted subdirs. I can get file list using `s4cmd ls`:

```
$ s4cmd ls s3:///*/*.conf
2019-02-23 02:01 0 s3:///dir1/
2019-04-04 17:42 525 s3:///dir1/main.conf
2019-04-04 17:42 755 s3:///dir1/test.conf
2019-04-04 17:42 525 s3:///dir2/test.conf
2019-04-04 17:42 755 s3:///dir2/main.conf
2019-02-26 04:06 0 s3:///dir3/
2019-04-04 17:43 525 s3:///dir3/main.conf
2019-04-04 17:43 755 s3:///dir2/test.conf
```

I dont know why, but `dir2` is not shown in the `output`.

I've tried to download all these files using: `s4cmd get`, `s4cmd sync` and `s4cmd dsync` commands and their `-r` counterparts, but all of them either provide undesired results or produce errors:

```
$ # ignores directory structure
$ s4cmd get s3:///*/*.conf .
omitting directory "s3:///dir1/".
omitting directory "s3:///dir3/".
s3:///dir3/main.conf => ./main.conf
s3:///dir2/test.conf => ./test.conf
s3:///dir3/test.conf => ./test.conf
s3:///dir1/main.conf => ./main.conf
s3:///dir1/test.conf => ./test.conf
s3:///dir2/main.conf => ./main.conf
```

```
$ # produces errors trying to get required files both to proper dirs and to sync root
$ # also produces errors denoting file names from assumed not be interfered
$ # similar results with s4cmd sync and s4cmd sync -r
$ s4cmd get -r s3:///*/*.conf
[Runtime Failure] File already exists: ./dir1/.
Exception in thread Thread-3:ad(s)]
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/bin/s4cmd", line 530, in run
fail('[Runtime Failure] ', exc_info=e)
File "/usr/bin/s4cmd", line 192, in fail
clean_tempfiles()
File "/usr/bin/s4cmd", line 218, in clean_tempfiles
for fn in TEMP_FILES:
RuntimeError: Set changed size during iteration

s3:///dir2/main.conf => ./main.conf
s3:///dir1/test.conf => ./dir1/test.conf
s3:///dir2/test.conf => ./test.conf
s3:///dir1/test.conf => ./test.conf
[Runtime Failure] Download Failure: , Source: s3:///dir1/subdir/.
s3:///dir1/main.conf => ./dir1/mainconf
s3:///dir1/main.conf => ./main.conf
[Runtime Failure] File already exists: ./test.conf
[Runtime Failure] File already exists: ./main.conf
s3:///dir3/test.conf => ./dir3/test.conf
[Runtime Failure] File already exists: ./dir3/.
s3:///dir3/subdir/ => ./dir3/subdir
Exception in thread Thread-11:ad(s)]
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/bin/s4cmd", line 530, in run
fail('[Runtime Failure] ', exc_info=e)
File "/usr/bin/s4cmd", line 192, in fail
clean_tempfiles()
File "/usr/bin/s4cmd", line 220, in clean_tempfiles
os.unlink(fn)
OSError: [Errno 2] No such file or directory: './dir3/subdir-m4iyx9a5s3zbnlq.tmp'

s3:///dir3/main.conf => ./dir3/main.conf
[OSError] 2: No such file or directory
[OSError] 2: No such file or directory
[Thread Failure] [Errno 2] No such file or directory: './dir1/subdir/-mufsq9gd640jr71.tmp'
```

```
$ # produces errors and doesnt do anything sensible either with or without -r flag
$ s4cmd dsync s3:///*/*.conf
Runtime Failure] The obj "*/*.conf/../../dir3" does not exists.
[Runtime Failure] The obj "*/*.conf/../../dir3/main.conf" does not exists.
[Runtime Failure] The obj "*/*.conf/../../dir3/test.conf" does not exists.
[Runtime Failure] The obj "*/*.conf/../../dir2/main.conf" does not exists.
[Runtime Failure] The obj "*/*.conf/../../dir2/test.conf" does not exists.
[Runtime Failure] The obj "*/*.conf/../../dir1" does not exists.
[Runtime Failure] The obj "*/*.conf/../../dir1/main.conf" does not exists.
[Runtime Failure] The obj "*/*.conf/../../dir1/test.conf" does not exists.
[Thread Failure] The obj "*/*.conf/../../dir1/test.conf" does not exists.
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the nested wildcard cases with the s4cmd ls, get, sync, and dsync entry points against a matching S3 layout. Trace how wildcard results and directory targets are handled, including recursive downloads, and verify that only the requested *.conf files are transferred while their nested subdirectories remain untouched and no duplicate-path or cleanup errors occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.