Improve encoding/decoding performance
- Dominant language
- JavaScript
- Stars
- 171
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Currently, Dbux encoding/decoding is its biggest performance bottleneck.
* [ ] Change to custom encoding/decoding → use type information to replace current `array-of-dictionary` approach
* → Should give many X of speed-up.
* NOTE currently, all data is encoded as dictionaries, of which we can end up having millions and even more.
* Currently: an array-of-object is actually stored as an `array-of-dictionary`, such as `[{ veryLongProp1: 1, anotherPropHere: 2 }, { veryLongProp1: 3, anotherPropHere: 4 }, ...]`
* It should become something like `{ props: [ 'veryLongProp, anotherPropHere' ], data: [1, 2, 3, 4, ...] }`
* [ ] Profile the entire data transmission stack: which part takes how long?
* [ ] Don't `waitForAck` on client
## Some More Notes
* We are using msgpack ([code here](https://github.com/Domiii/dbux/blob/master/dbux-common/src/msgpackParser.js)).
* On Node, one could use [v8.serialize](https://nodejs.org/api/v8.html#v8serializevalue).
* Some performance comparisons can be found [here](https://github.com/babel/babel/issues/5667#issuecomment-1103243965)
* It's max size is generally 4GB on 64 bit systems.
Contributor guide
Research direction
Start with dbux-common/src/msgpackParser.js and profile the entire data transmission stack to identify where time is spent. Review the msgpack implementation and the linked Node v8.serialize option, then determine the scope of custom encoding and client acknowledgement changes. Done means the bottlenecks are measured and the encoding/decoding and acknowledgement behavior meet the issue's performance goals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100