awslabs / awslabs/amazon-dynamodb-tools
[verb] `export-replay` and `export-unplay` for processing incremental exports
- Dominant language
- Python
- Stars
- 181
- Forks
- 50
- Avg merge
- 4h 11m
- Merged PRs (30d)
- 36
Description
A useful verb would let you read an incremental export and play that sequence of events forward or backward on a table.
Backward: Would let you undo actions during the specified time period. If you did a bad load, you could do an incremental export of the load window, run this verb, and it would undo all those changes. Basically in the incremental export if it saw a New Image and no Old Image it would know that was an insert and it would delete the item to put it back to its original state. If it saw an Old Image and no New Image it would know that was a delete and it would write to put back the item. If it was an Old Image and New Image it would know that was an update and it would put back the Old Image.
Ideally there'd be some sort of "filter" applied so a user could not undo ALL the actions but only ones satisfying some criteria. We might call out to a python function to yes/no each item being rolled back, default to roll back everything, but a smart python script could only do certain PK values or whatever.
Forward: Same idea but the inverse. This would be helpful if you wanted to apply a series of incremental exports to a table to catch it up to another table. You write a full export then series of incremental exports to copy data to a second region. There's a need to play time forward in the second region when rehydrating the table based on the export series.
With "filtering" added, the playing forward could do partial replication.
Possibly there could be a transform step as well.
Need to decide the right verb names. And if this is one verb or two.
```
./bulk export-replay --s3-location s3://foo --table target --filter foo.py
./bulk export-unplay --s3-location s3://foo --table target --filter foo.py
```
Contributor guide
Assessment
This issue has not been assessed yet.