awslabs / awslabs/amazon-dynamodb-tools

[verb] `copy` - to copy from one table to another with a hook to transform

Open
#93 2 comments 0 reactions 0 assignees View on GitHub
bulk_executor bulk-verb enhancement help wanted
Dominant language
Python
Stars
181
Forks
50
Avg merge
4h 11m
Merged PRs (30d)
36

Description

If you want to move data from one table to another and possibly manipulate the items as they go, that's something we could provide with a copy verb.

```
./bulk copy --from t1 --to t2 --transformer trans.py
```

The transformer could limit what gets sent, reshape it, redact data, etc.

If there's no need for the transform, a backup/restore would be sufficient, so this is really about a transformational copy. A better name might be `transform` but that feels like it would be in-place.

An export, manipulate the content in S3, and import to a new table is another option. This would be more straightforward.

Contributor guide

Open the contributing guide

Research direction

Start with the proposed `./bulk copy --from t1 --to t2 --transformer trans.py` entry point and compare it with the stated export, S3 manipulation, and import alternative. Done means the project has a decided transformational-copy design, including how the transformer participates; the issue names no implementation files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.