aws / aws/aws-cli

Copying data to s3 from a named pipe using process substitution fails

Open
#6,145 5 comments 5 reactions 0 assignees View on GitHub
feature-request p3 s3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

- [x] I've gone though the [User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) and the [API reference](https://docs.aws.amazon.com/cli/latest/reference/)
- [x] I've searched for [previous similar issues](https://github.com/aws/aws-cli/issues) and didn't find any solution

This is possible:
```
$ cp <(echo "Test") test.txt
```

This is possible:
```
$ aws s3 cp s3://my-bucket/test.txt >(cat > test.txt)
```

This is _not_ possible:
```
$ aws s3 cp <(echo "Test") s3://my-bucket/test.txt
warning: Skipping file /dev/fd/63. File is character special device, block special device, FIFO, or socket.
```

and yet, this is possible:
```
$ echo Test | aws s3 cp - s3://my-bucket/test.txt
```

This is inconsistent behaviour and thus not aligned with user expectations. The docs say FIFO queues are skipped, but that's clearly not always the case since `-` works. Strangely, even using `--include '*'` doesn't help matters.

**SDK version number**
aws-cli/1.19.1 Python/3.7.3 Linux/5.10.0-0.bpo.5-amd64 botocore/1.20.0

**Platform/OS/Hardware/Device**
Debian GNU/Linux 10, x86_64

**To Reproduce (observed behavior)**
`$ aws s3 cp <(echo "Test") s3://my-bucket/test.txt`

**Expected behavior**
I expect `s3://my-bucket/test.txt` should exist as an S3 object with the text "Test".

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the process-substitution command and compare its handling with `echo Test | aws s3 cp - s3://my-bucket/test.txt`. Trace the S3 copy entry point for `/dev/fd/63` versus stdin; done means the named-pipe input uploads as an S3 object while the documented special-file behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.