handshake-org / handshake-org/hsd

Namestate optimization

Open
#620 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.1k
Forks
306
PR merge metrics
No merged PRs in 30d

Description

We currently have two blockchain optimizations in review:
- Flat file block storage (https://github.com/handshake-org/hsd/pull/612)
- "Assume valid" checkpoints (https://github.com/handshake-org/hsd/pull/597)

Somewhat surprisingly, we are not seeing a very dramatic performance improvement from either of these, even when combined into a single branch. They are both huge boosts for Bitcoin and its possible that we will see a more dramatic benefit from them as the Handshake blockchain gets bigger.

However, it's occurred to me that in addition to optimizing signature verification and data storage and other Bitcoin-y stuff, what we need to really look at is how Namestate is processed during blockchain sync.

The first thing we can try is simply replace the JavaScript Urkel tree module with the new library written in C: https://github.com/chjj/liburkel

But I think even after data is fetched from Urkel there are performance improvements we can investigate:

### Caching

When verifying a block, the `CoinView` object gets `NameState` from urkel and caches them [here](https://github.com/handshake-org/hsd/blob/98a6491cdbfb173b4834a892b9bd55b6839cadbf/lib/covenants/view.js#L36-L57). So even if a block has ten bids for the same name, that name is only pulled from the database once. However, the NameState for a name _does not change_ until the reveal period begins -- thats `(5 * 144) + 36` blocks that we can cache a NameState after processing the `OPEN`. I believe we can be clever about how NameState is cached in chain verification and save some reads from disk. Even during the reveal phase when NameState _does_ get updated (if a reveal has a higher bid) we could theoretically cache those updates as well for `10 * 144` blocks and avoid unnecessary reads (although a higher bid value would still have to be written to disk)

### Serialization

In bcoin and hsd there is object called a [`MemBlock`](https://github.com/handshake-org/hsd/blob/master/lib/primitives/memblock.js) and I think we can use a similar model for NameState.

My first thought about this was, especially during the bidding phase, the only thing we need to know about a name is its `height` and maybe some of the flags like `revoked`. So, do we need to `NameState.decode(raw)` for every name we verify? Do we need JavaScript object instantiation and processing for the entire namestate, for every name?

What would be awesome is if we could just grab the raw buffer from the Urkel tree, and read the `U32` height directly from the buffer without decoding anything else. The biggest problem with this idea is unfortunately the NameState serialization isn't optimized for this:

https://github.com/handshake-org/hsd/blob/98a6491cdbfb173b4834a892b9bd55b6839cadbf/lib/covenants/namestate.js#L572-L580

There are a few variable-length fields in the buffer _before_ the height. We'd have to read the name length to know where the data length is, then read the data length to know where the height is. That's not terrible and still might be an optimization but its not as clean as could be if NameState had a fixed-length "header". Re-serializing namestates in Urkel would be a super-brutal hardfork since the tree root committed in each block is based on the serialized namestate of each name in the tree.. and the NameState object has no version number to iterate :-(

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.