Copying data to s3 from a named pipe using process substitution fails
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
- [x] I've gone though the [User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) and the [API reference](https://docs.aws.amazon.com/cli/latest/reference/)
- [x] I've searched for [previous similar issues](https://github.com/aws/aws-cli/issues) and didn't find any solution
This is possible:
```
$ cp <(echo "Test") test.txt
```
This is possible:
```
$ aws s3 cp s3://my-bucket/test.txt >(cat > test.txt)
```
This is _not_ possible:
```
$ aws s3 cp <(echo "Test") s3://my-bucket/test.txt
warning: Skipping file /dev/fd/63. File is character special device, block special device, FIFO, or socket.
```
and yet, this is possible:
```
$ echo Test | aws s3 cp - s3://my-bucket/test.txt
```
This is inconsistent behaviour and thus not aligned with user expectations. The docs say FIFO queues are skipped, but that's clearly not always the case since `-` works. Strangely, even using `--include '*'` doesn't help matters.
**SDK version number**
aws-cli/1.19.1 Python/3.7.3 Linux/5.10.0-0.bpo.5-amd64 botocore/1.20.0
**Platform/OS/Hardware/Device**
Debian GNU/Linux 10, x86_64
**To Reproduce (observed behavior)**
`$ aws s3 cp <(echo "Test") s3://my-bucket/test.txt`
**Expected behavior**
I expect `s3://my-bucket/test.txt` should exist as an S3 object with the text "Test".
Contributor guide
Research direction
Start by reproducing the process-substitution command and compare its handling with `echo Test | aws s3 cp - s3://my-bucket/test.txt`. Trace the S3 copy entry point for `/dev/fd/63` versus stdin; done means the named-pipe input uploads as an S3 object while the documented special-file behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100