mapbox / mapbox/dynamodb-replicator

incremental backup and incremental backfill generate different file names

Open
#90 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
131
Forks
46
PR merge metrics
No merged PRs in 30d

Description

Hi there!

First off, great library. It's super useful and a much better/simpler option (for me) than the whole EMR/Datapipeline situation.

I have this simple lambda function that is subscribed to the tables I want to update:
(the bucket, region, and prefix are set as env variables in the lambda function)
```js
var replicator = require('dynamodb-replicator')
module.exports.streaming = (event, context, callback) => {
return replicator.backup(event, callback)
}
```
Then I ran the backfill by importing `dynamodb-replicator/s3-backfill` and passing it a config object.

However, I noticed that when records get updated via the stream/lambda function, they are written to a different file from the one created by the backfill.

I see that the formula for generating filenames is slightly different.

```js
\\backfilll
var id = crypto.createHash('md5')
.update(Dyno.serialize(key))
.digest('hex');

\\backup
var id = crypto.createHash('md5')
.update(JSON.stringify(change.dynamodb.Keys))
.digest('hex');
```
https://github.com/mapbox/dynamodb-replicator/blob/master/s3-backfill.js#L46-L48
https://github.com/mapbox/dynamodb-replicator/blob/master/index.js#L130-L132

Does this make any practical difference? Should the restore function work regardless?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the filename-generation code at lines 46-48 of s3-backfill.js and lines 130-132 of index.js, then inspect the restore implementation to determine whether the differing hashes matter. Verify restore behavior for files produced by both paths; done means the mismatch is resolved or its impact is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript, node.js
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.