Faster way to get block with prevouts in JSON-RPC
- Dominant language
- C++
- Stars
- 90.2k
- Forks
- 39.4k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 88
Description
I often need to process the whole blockchain (or a large part of it) using an external script/program, for which I need blocks with prevout information included. However, the only current way to get that is `getblock 3`, which includes a lot of potentially unnecessary data and is quite slow, mainly (based on my experiments) because of `UniValue` overhead and descriptor inferring.
I benchmarked current master, retrieving 1000 blocks sequentially starting at block 840000, with different `verbosity` parameters:
|benchmark|result|
|-|-|
|getblock (verbosity=0)|16.189s ± 1.165s|
|getblock (verbosity=1)|31.975s ± 1.014s|
|getblock (verbosity=2)|352.487s ± 1.636s|
|getblock (verbosity=3)|473.375s ± 2.280s|
As you can see, verbosity=3 is around 30 times slower than verbosity=0. It seems obvious that a faster way of getting blocks with prevout information is feasible.
Potential solutions that come to mind:
- Creating a new RPC call for undo data, say `getblockundo`. This would be perfect for my needs, but it would require making the undo data serialization format non-internal (not sure if this would be a problem, as IIRC it hasn't changed in many years).
- Creating a new verbosity level for `getblock` that would only provide the minimum amount of data necessary (i.e. no addresses, descriptors, ASM scripts, TXIDs/WTXIDs etc.) while still providing prevouts. This would be better than nothing but would still leave a lot of performance on the table because of `UniValue` overhead.
Contributor guide
Research direction
Start at the getblock JSON-RPC entry point and compare the verbosity=0, 1, 2, and 3 paths, especially prevout handling and the reported UniValue and descriptor-inference costs. Reproduce the 1,000-block benchmark from height 840000; done requires an agreed API or verbosity path that returns prevouts with materially less overhead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100