awslabs / awslabs/amazon-dynamodb-tools

[bulk] Optionally let users choose the S3 output location for large-output commands (find/sql/diff)

Open
#281 0 comments 0 reactions 0 assignees View on GitHub
bulk_executor help wanted
Dominant language
Python
Stars
181
Forks
50
Avg merge
4h 11m
Merged PRs (30d)
36

Description

### Idea

Large-output commands (`find`, `sql`, and — once aligned per #86 — `diff`) write their full output to a **fixed** S3 location: `s3://{bootstrap-bucket}/output/{JOB_RUN_ID}`. There is currently no way for a user to choose where that output goes.

Aligning `diff` with the find convention removes its `--s3` flag (its only S3-related knob), so after that change none of the large-output commands expose any control over the S3 destination.

### Proposal (not doing now)

Consider adding an optional flag to set a custom S3 output location for the large-output commands — e.g. `--s3-output s3://my-bucket/my-prefix/`. This would make outputs easier to consume in a pipeline (a `find` writing somewhere a subsequent `load` reads from — see #184) and gives users control over lifecycle/retention of the output.

Deferring for now; filing so it isn't lost. Relates to #180 (structured output convention) and #184 (find→load round-trip).

Contributor guide

Open the contributing guide

Research direction

Start with the find, sql, and diff command entry points and trace how their large output is written to the fixed bootstrap-bucket S3 location. Review the related issues #86, #180, and #184 before deciding the flag's scope and behavior; the work is done when users can optionally choose a destination consistently across the large-output commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.