mapbox / mapbox/dynamodb-replicator
dyno failing to restore incremental backup?
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 131
- Forks
- 46
- PR merge metrics
- No merged PRs in 30d
Description
simple test case with a small table:
```
ENV=dev
TABLE=dsrtest2
. config.env.$ENV
bin/incremental-backfill.js $AWS_REGION/$TABLE s3://$BackupBucket/$BackupPrefix
bin/incremental-snapshot.js s3://$BackupBucket/$BackupPrefix/$TABLE s3://$BackupBucket/${TABLE}-snapshot
s3print s3://$BackupBucket/${TABLE}-snapshot | dyno put $AWS_REGION/dsr-test-restore-$TABLE
```
```
%% sh test-backup.sh
12 - 11.89/s[Fri, 09 Dec 2016 23:54:59 GMT] [info] [incremental-snapshot] Starting snapshot from s3://dsr-ddb-rep-testing/testprefix/dsrtest2 to s3://dsr-ddb-rep-testing/dsrtest2-snapshot
[Fri, 09 Dec 2016 23:55:01 GMT] [info] [incremental-snapshot] Starting upload of part #0, 0 bytes uploaded, 12 items uploaded @ 6.26 items/s
[Fri, 09 Dec 2016 23:55:01 GMT] [info] [incremental-snapshot] Uploaded snapshot to s3://dsr-ddb-rep-testing/dsrtest2-snapshot
[Fri, 09 Dec 2016 23:55:01 GMT] [info] [incremental-snapshot] Wrote 12 items and 148 bytes to snapshot
undefined:1
�
^
SyntaxError: Unexpected token in JSON at position 0
at Object.parse (native)
at Function.module.exports.deserialize (/Users/draistrick/git/github/dynamodb-replicator/node_modules/dyno/lib/serialization.js:49:18)
at Transform.Parser.parser._transform (/Users/draistrick/git/github/dynamodb-replicator/node_modules/dyno/bin/cli.js:94:25)
at Transform._read (_stream_transform.js:167:10)
at Transform._write (_stream_transform.js:155:12)
at doWrite (_stream_writable.js:307:12)
at writeOrBuffer (_stream_writable.js:293:5)
at Transform.Writable.write (_stream_writable.js:220:11)
at Stream.ondata (stream.js:31:26)
at emitOne (events.js:96:13)
```
Next step would be to diff the two tables - but the pipe to dyno fails. I've tried 1.0.0 and 1.3.0 with the same result.
What data format is dyno expecting? The file on s3 (tried multiple tables including real data tables) is a binary blob?
```
cheese:~%% aws --region=us-west-2 s3 cp s3://dsr-ddb-rep-testing/dsrtest-snapshot -
m�1�
��ߠl�EG�EB�uL0\�Tuq�ݵ#������$L�6�/8�%Z�r�[d�p
���5h)��X�ֻ�j�ƪ�
ۘ��&�WJ'❑��`�T�������
cheese:~%%
```
So maybe this is a problem with backfill? or I'm missing something? :)
2016-12-09 18:54:35 149 dsrtest-snapshot
2016-12-09 18:55:01 148 dsrtest2-snapshot
2016-12-09 18:37:20 1428 receipt_log_dev-01-snapshot
2016-12-09 18:53:15 13457328 showdownlive_dev-01-snapshot
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the pipeline in the issue with bin/incremental-backfill.js, bin/incremental-snapshot.js, and s3print. Read node_modules/dyno/lib/serialization.js and node_modules/dyno/bin/cli.js where the JSON parse failure occurs, then compare the snapshot output with dyno's expected input format. Done means the generated snapshot can be piped through dyno put successfully and the restored table can be compared with the source.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript, nodejs
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100