Possible data loss when importing into DynamoDB!
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 79
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
While trying to use the cli to import data into DynamoDB I wondered why no data appeared in DynamoDB.
After some digging I noticed that [`aggregator._flush`](https://github.com/mapbox/dyno/blob/master/bin/cli.js#L162-L165) isn't called when piping data into `dyno`. That causes up to 25 records not to be imported into DynamoDB and happens for `put` as well as for `import`!
In my case my test data had less than 25 records, that's why I noticed it.
Tested on Debian/unstable with NodeJS 7.10.0 & 8.1.3.
The internet seems to suggest (https://github.com/nodejs/node/issues/53) that a `.push(null)` at the end of the stream is necessary, but as I'm not really familiar with NodeJS I don't know if that applies to reading from `stdin` as well.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in bin/cli.js at aggregator._flush around lines 162-165 and reproduce the issue by piping fewer than 25 records into dyno for both put and import. Trace stdin completion and confirm the change causes all piped records to reach DynamoDB, including batches smaller than 25.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript, node.js
- Domain
- cli, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100