GoogleCloudPlatform / GoogleCloudPlatform/gsutil
cp command -I flag is ignored when writing to stdout
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
When using the cp command `-I` flag while sending the output to stdout, the command exits without doing anything:
```bash
# local file to showcase the problem, the same behavior occurs when listing files from gcs
$ echo test > /tmp/gsutil_test
$ ls /tmp/gsutil_test | gsutil cp -I -
$ echo $?
0
# the command works when given a destination:
$ ls /tmp/gsutil_test | gsutil cp -I /tmp/gsutil_test.out && cat /tmp/gsutil_test.out
Copying file:///tmp/gsutil_test...
/ [1 files][ 5.0 B/ 5.0 B]
Operation completed over 1 objects/5.0 B.
test
# copying to stdout works when -I isn't used
$ gsutil cp /tmp/gsutil_test -
test
$
```
The expected outcome of `ls /tmp/gsutil_test | gsutil cp -I -` is to have `test` echoed on stdout.
I believe the problem comes from the fact that the ["stdout/pipe"](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/commands/cp.py#L1118) check is done before checking to see if the [-I flag was used](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/commands/cp.py#L1127).
Moving the "-I flag" check above means we could use `src_url_strs` in the `cat_helper.CatHelper(self).CatUrlStrings()` call.
There are a few problems though, as `CatUrlStrings()` expects a `List[str]`, while `src_url_strs` is a `List[Iterator[str]]`, and changing its signature is not trivial since the code is shared with the `gsutil cat` command.
I can have a look at it and try and submit a PR, if you think this is worth changing.
-----------------------
Versions:
```
$ gcloud version
Google Cloud SDK 297.0.1
alpha 2020.06.17
beta 2020.06.17
bq 2.0.58
core 2020.06.17
gsutil 4.51
kubectl 2020.06.17
```
Contributor guide
Research direction
Start in gslib/commands/cp.py at the stdout/pipe check and the -I handling around the referenced lines. Trace how src_url_strs reaches cat_helper.CatHelper(self).CatUrlStrings(), paying attention to the Iterator-versus-List mismatch and shared gsutil cat behavior. Reproduce with `ls /tmp/gsutil_test | gsutil cp -I -`; done means the file contents are written to stdout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100