mapbox / mapbox/dynamodb-replicator
incremental backup and incremental backfill generate different file names
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 131
- Forks
- 46
- PR merge metrics
- No merged PRs in 30d
Description
Hi there!
First off, great library. It's super useful and a much better/simpler option (for me) than the whole EMR/Datapipeline situation.
I have this simple lambda function that is subscribed to the tables I want to update:
(the bucket, region, and prefix are set as env variables in the lambda function)
```js
var replicator = require('dynamodb-replicator')
module.exports.streaming = (event, context, callback) => {
return replicator.backup(event, callback)
}
```
Then I ran the backfill by importing `dynamodb-replicator/s3-backfill` and passing it a config object.
However, I noticed that when records get updated via the stream/lambda function, they are written to a different file from the one created by the backfill.
I see that the formula for generating filenames is slightly different.
```js
\\backfilll
var id = crypto.createHash('md5')
.update(Dyno.serialize(key))
.digest('hex');
\\backup
var id = crypto.createHash('md5')
.update(JSON.stringify(change.dynamodb.Keys))
.digest('hex');
```
https://github.com/mapbox/dynamodb-replicator/blob/master/s3-backfill.js#L46-L48
https://github.com/mapbox/dynamodb-replicator/blob/master/index.js#L130-L132
Does this make any practical difference? Should the restore function work regardless?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the filename-generation code at lines 46-48 of s3-backfill.js and lines 130-132 of index.js, then inspect the restore implementation to determine whether the differing hashes matter. Verify restore behavior for files produced by both paths; done means the mismatch is resolved or its impact is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript, node.js
- Domain
- backend, cloud, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100