aws / aws/aws-cli

Support piping DynamoDB query / scan output to another command

Open
#6,283 3 comments 0 reactions 0 assignees View on GitHub
dynamodb feature-request p3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
When I use the AWS CLI to query or scan a DynamoDB table, I am unable to pipe that output to another command (effectively) because the JSON structure of the output requires the output to be 100% complete before another command can process it. If you need to whip up a quick-and-dirty 'query this table for data, and send each row to this other command' type job, you can't effectively do so if the output is thousands, tens of thousands, or millions of lines - the entire JSON output will be buffered, resulting in extremely slow processing and a huge load on both the CLI itself and the next command in your pipeline to process that giant JSON.

**Describe the solution you'd like**
Line-delimited JSON for datasets such as DynamoDB queries, scans, S3 lists, etc. Then each line can be output from the CLI as soon as it's processed, and the next command in the pipeline can process that line without waiting for the entire dataset to be complete.

**Describe alternatives you've considered**
The alternative is writing my own scripts with the SDK, removing the flexibility and speed of just using the CLI for one-off tasks.

**Additional context**
See also https://github.com/aws/aws-cli/issues/4703#issuecomment-657343054

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.