bitcoin / bitcoin/bitcoin

Faster way to get block with prevouts in JSON-RPC

Open
#30,495 10 comments 2 reactions 0 assignees View on GitHub
Block storage Feature RPC/REST/ZMQ
Dominant language
C++
Stars
90.2k
Forks
39.4k
Avg merge
3d 8h
Merged PRs (30d)
88

Description

I often need to process the whole blockchain (or a large part of it) using an external script/program, for which I need blocks with prevout information included. However, the only current way to get that is `getblock 3`, which includes a lot of potentially unnecessary data and is quite slow, mainly (based on my experiments) because of `UniValue` overhead and descriptor inferring.

I benchmarked current master, retrieving 1000 blocks sequentially starting at block 840000, with different `verbosity` parameters:

|benchmark|result|
|-|-|
|getblock (verbosity=0)|16.189s ± 1.165s|
|getblock (verbosity=1)|31.975s ± 1.014s|
|getblock (verbosity=2)|352.487s ± 1.636s|
|getblock (verbosity=3)|473.375s ± 2.280s|

As you can see, verbosity=3 is around 30 times slower than verbosity=0. It seems obvious that a faster way of getting blocks with prevout information is feasible.

Potential solutions that come to mind:
- Creating a new RPC call for undo data, say `getblockundo`. This would be perfect for my needs, but it would require making the undo data serialization format non-internal (not sure if this would be a problem, as IIRC it hasn't changed in many years).
- Creating a new verbosity level for `getblock` that would only provide the minimum amount of data necessary (i.e. no addresses, descriptors, ASM scripts, TXIDs/WTXIDs etc.) while still providing prevouts. This would be better than nothing but would still leave a lot of performance on the table because of `UniValue` overhead.

Contributor guide

Open the contributing guide

Research direction

Start at the getblock JSON-RPC entry point and compare the verbosity=0, 1, 2, and 3 paths, especially prevout handling and the reported UniValue and descriptor-inference costs. Reproduce the 1,000-block benchmark from height 840000; done requires an agreed API or verbosity path that returns prevouts with materially less overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.