awslabs / awslabs/amazon-dynamodb-tools

[verb] `signature` - for comparing tables across regions or time

Open
#98 0 comments 0 reactions 0 assignees View on GitHub
bulk_executor bulk-verb enhancement help wanted
Dominant language
Python
Stars
181
Forks
50
Avg merge
4h 11m
Merged PRs (30d)
36

Description

We could do a "signature" verb that gave, for example, a hash of the table or parts of a table. This would allow efficient comparisons of two tables across regions or time. Similar to how you use a signature to check two files match, but here it's tables.

Regions: You could compare two endpoints of a global table without copying any table data out of region (as you'd have to with a `diff`).

Time: You can store the signature of a table before a backup, then compare the signature later after a restore.

One way to build this would be to do a segmented scan with X segments, generate a hash of each segment, then a hash of the hashes. If the hash of hashes doesn't match, you can know in what segment the mismatch was. The output would be the X+1 hashes (to S3 probably).

We'd need a hash algorithm that could take a stream of items serialized in some very deterministic way to build the hash up.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the existing verb implementation and DynamoDB scan behavior, then review how S3 outputs are handled. Done should mean a decided signature design, including deterministic item serialization, segmented results, and comparison behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.