oasisprotocol / oasisprotocol/oasis-core

Reduce delay in roothash commit processing

Open
#1,732 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

c:consensus/cometbft c:performance c:roothash c:runtime/compute epic
Dominant language
Go
Stars
369
Forks
151
Avg merge
1d 3h
Merged PRs (30d)
6

Description

Since Tendermint blindly schedules when blocks are proposed, the following can happen, causing unnecessary delay while processing batches:

2019-05-21T10:08:29.299999503Z commit called on node 1
2019-05-21T10:08:29.299999503Z tx roothash commit from node 1 (->h=25)
2019-05-21T10:08:29.30041168Z  enter propose (h=25)
2019-05-21T10:08:29.382282441Z tx roothash commit from node 2 (->h=26)
2019-05-21T10:08:29.385026717Z tx roothash commit from node 3 (->h=26)
2019-05-21T10:08:29.389549228Z tx roothash commit from node 4 (->h=26)
2019-05-21T10:08:29.719338595Z executed block (h=25)
2019-05-21T10:08:29.724729696Z enter propose (h=26)
2019-05-21T10:08:30.20912014Z  executed block (h=26)

Basically the first commitment causes a new Tendermint block to be proposed and executed, containing only the one transaction corresponding the the first roothash commit. The subsequent three commits that arrive just 80ms later are put into the next block and the whole roothash round is delayed by ~500ms.

Possible solutions:

  1. Broadcast commits between compute committee members via p2p. Then each committee member can push them in bulk to Tendermint once it sees enough of them or a timeout elapses (duplicate transactions will be ignored by mempool anyway).
  2. Implement a smarter Tendermint mempool (possible in latest Tendermint develop branch as Mempool is now an interface) which allows ABCI apps to emit some scheduling information from CheckTx based on which it could wait "a bit" before releasing any transactions. This seems much more complex and may be less effective (as the mempool can also contain a bunch of non-roothash transactions).
  3. Something else.

If we decide to do this, I would be in favor of (1) as it keeps this functionality under our control and avoids the complexity of implementing (2).

In any case, these effects probably cannot be completely eliminated so as a mitigation we should make sure that:

  • We can process large batches of transactions to reduce the effect of Tendermint latency on throughput.
  • We implement something like batch pipelineing (#1581).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests. Start by reviewing roothash commit processing, Tendermint scheduling, and the related batch-pipeline discussion in #1581; compare the proposed P2P and Tendermint-mempool approaches. Done requires a decided, scoped mitigation with its batching and throughput effects defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
blockchain, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.