awslabs / awslabs/amazon-dynamodb-tools
[bulk] Optionally let users choose the S3 output location for large-output commands (find/sql/diff)
- Dominant language
- Python
- Stars
- 181
- Forks
- 50
- Avg merge
- 4h 11m
- Merged PRs (30d)
- 36
Description
### Idea
Large-output commands (`find`, `sql`, and — once aligned per #86 — `diff`) write their full output to a **fixed** S3 location: `s3://{bootstrap-bucket}/output/{JOB_RUN_ID}`. There is currently no way for a user to choose where that output goes.
Aligning `diff` with the find convention removes its `--s3` flag (its only S3-related knob), so after that change none of the large-output commands expose any control over the S3 destination.
### Proposal (not doing now)
Consider adding an optional flag to set a custom S3 output location for the large-output commands — e.g. `--s3-output s3://my-bucket/my-prefix/`. This would make outputs easier to consume in a pipeline (a `find` writing somewhere a subsequent `load` reads from — see #184) and gives users control over lifecycle/retention of the output.
Deferring for now; filing so it isn't lost. Relates to #180 (structured output convention) and #184 (find→load round-trip).
Contributor guide
Research direction
Start with the find, sql, and diff command entry points and trace how their large output is written to the fixed bootstrap-bucket S3 location. Review the related issues #86, #180, and #184 before deciding the flag's scope and behavior; the work is done when users can optionally choose a destination consistently across the large-output commands.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100