GoogleCloudPlatform / GoogleCloudPlatform/gsutil

cp command -I flag is ignored when writing to stdout

Open
#1,009 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

When using the cp command `-I` flag while sending the output to stdout, the command exits without doing anything:

```bash
# local file to showcase the problem, the same behavior occurs when listing files from gcs
$ echo test > /tmp/gsutil_test
$ ls /tmp/gsutil_test | gsutil cp -I -
$ echo $?
0

# the command works when given a destination:
$ ls /tmp/gsutil_test | gsutil cp -I /tmp/gsutil_test.out && cat /tmp/gsutil_test.out
Copying file:///tmp/gsutil_test...
/ [1 files][ 5.0 B/ 5.0 B]
Operation completed over 1 objects/5.0 B.
test

# copying to stdout works when -I isn't used
$ gsutil cp /tmp/gsutil_test -
test
$
```

The expected outcome of `ls /tmp/gsutil_test | gsutil cp -I -` is to have `test` echoed on stdout.

I believe the problem comes from the fact that the ["stdout/pipe"](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/commands/cp.py#L1118) check is done before checking to see if the [-I flag was used](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/commands/cp.py#L1127).

Moving the "-I flag" check above means we could use `src_url_strs` in the `cat_helper.CatHelper(self).CatUrlStrings()` call.
There are a few problems though, as `CatUrlStrings()` expects a `List[str]`, while `src_url_strs` is a `List[Iterator[str]]`, and changing its signature is not trivial since the code is shared with the `gsutil cat` command.

I can have a look at it and try and submit a PR, if you think this is worth changing.

-----------------------
Versions:
```
$ gcloud version
Google Cloud SDK 297.0.1
alpha 2020.06.17
beta 2020.06.17
bq 2.0.58
core 2020.06.17
gsutil 4.51
kubectl 2020.06.17
```

Contributor guide

Open the contributing guide

Research direction

Start in gslib/commands/cp.py at the stdout/pipe check and the -I handling around the referenced lines. Trace how src_url_strs reaches cat_helper.CatHelper(self).CatUrlStrings(), paying attention to the Iterator-versus-List mismatch and shared gsutil cat behavior. Reproduce with `ls /tmp/gsutil_test | gsutil cp -I -`; done means the file contents are written to stdout.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.