oasisprotocol / oasisprotocol/oasis-core
Reduce delay in roothash commit processing
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 369
- Forks
- 151
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
Since Tendermint blindly schedules when blocks are proposed, the following can happen, causing unnecessary delay while processing batches:
2019-05-21T10:08:29.299999503Z commit called on node 1
2019-05-21T10:08:29.299999503Z tx roothash commit from node 1 (->h=25)
2019-05-21T10:08:29.30041168Z enter propose (h=25)
2019-05-21T10:08:29.382282441Z tx roothash commit from node 2 (->h=26)
2019-05-21T10:08:29.385026717Z tx roothash commit from node 3 (->h=26)
2019-05-21T10:08:29.389549228Z tx roothash commit from node 4 (->h=26)
2019-05-21T10:08:29.719338595Z executed block (h=25)
2019-05-21T10:08:29.724729696Z enter propose (h=26)
2019-05-21T10:08:30.20912014Z executed block (h=26)
Basically the first commitment causes a new Tendermint block to be proposed and executed, containing only the one transaction corresponding the the first roothash commit. The subsequent three commits that arrive just 80ms later are put into the next block and the whole roothash round is delayed by ~500ms.
Possible solutions:
- Broadcast commits between compute committee members via p2p. Then each committee member can push them in bulk to Tendermint once it sees enough of them or a timeout elapses (duplicate transactions will be ignored by mempool anyway).
- Implement a smarter Tendermint mempool (possible in latest Tendermint
developbranch as Mempool is now an interface) which allows ABCI apps to emit some scheduling information from CheckTx based on which it could wait "a bit" before releasing any transactions. This seems much more complex and may be less effective (as the mempool can also contain a bunch of non-roothash transactions). - Something else.
If we decide to do this, I would be in favor of (1) as it keeps this functionality under our control and avoids the complexity of implementing (2).
In any case, these effects probably cannot be completely eliminated so as a mitigation we should make sure that:
- We can process large batches of transactions to reduce the effect of Tendermint latency on throughput.
- We implement something like batch pipelineing (#1581).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests. Start by reviewing roothash commit processing, Tendermint scheduling, and the related batch-pipeline discussion in #1581; compare the proposed P2P and Tendermint-mempool approaches. Done requires a decided, scoped mitigation with its batching and throughput effects defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- blockchain, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100