Domiii / Domiii/dbux

Improve encoding/decoding performance

Open
#728 0 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
JavaScript
Stars
171
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Currently, Dbux encoding/decoding is its biggest performance bottleneck.

* [ ] Change to custom encoding/decoding → use type information to replace current `array-of-dictionary` approach
* → Should give many X of speed-up.
* NOTE currently, all data is encoded as dictionaries, of which we can end up having millions and even more.
* Currently: an array-of-object is actually stored as an `array-of-dictionary`, such as `[{ veryLongProp1: 1, anotherPropHere: 2 }, { veryLongProp1: 3, anotherPropHere: 4 }, ...]`
* It should become something like `{ props: [ 'veryLongProp, anotherPropHere' ], data: [1, 2, 3, 4, ...] }`
* [ ] Profile the entire data transmission stack: which part takes how long?
* [ ] Don't `waitForAck` on client

## Some More Notes
* We are using msgpack ([code here](https://github.com/Domiii/dbux/blob/master/dbux-common/src/msgpackParser.js)).
* On Node, one could use [v8.serialize](https://nodejs.org/api/v8.html#v8serializevalue).
* Some performance comparisons can be found [here](https://github.com/babel/babel/issues/5667#issuecomment-1103243965)
* It's max size is generally 4GB on 64 bit systems.

Contributor guide

Open the contributing guide

Research direction

Start with dbux-common/src/msgpackParser.js and profile the entire data transmission stack to identify where time is spent. Review the msgpack implementation and the linked Node v8.serialize option, then determine the scope of custom encoding and client acknowledgement changes. Done means the bottlenecks are measured and the encoding/decoding and acknowledgement behavior meet the issue's performance goals.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.