Support piping DynamoDB query / scan output to another command
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
When I use the AWS CLI to query or scan a DynamoDB table, I am unable to pipe that output to another command (effectively) because the JSON structure of the output requires the output to be 100% complete before another command can process it. If you need to whip up a quick-and-dirty 'query this table for data, and send each row to this other command' type job, you can't effectively do so if the output is thousands, tens of thousands, or millions of lines - the entire JSON output will be buffered, resulting in extremely slow processing and a huge load on both the CLI itself and the next command in your pipeline to process that giant JSON.
**Describe the solution you'd like**
Line-delimited JSON for datasets such as DynamoDB queries, scans, S3 lists, etc. Then each line can be output from the CLI as soon as it's processed, and the next command in the pipeline can process that line without waiting for the entire dataset to be complete.
**Describe alternatives you've considered**
The alternative is writing my own scripts with the SDK, removing the flexibility and speed of just using the CLI for one-off tasks.
**Additional context**
See also https://github.com/aws/aws-cli/issues/4703#issuecomment-657343054
Contributor guide
Assessment
This issue has not been assessed yet.