Disk IO pipelining in UTxO-HD

This is a summary of the discussion held in Consensus Office Hours on 29th of April 2025.

With UTxO-HD there is an opportunity to start reading the values from the tables for the coming blocks in a thread separate from the Chain selection thread.

• Chain selection selects a candidate in the volatile DB and then validates it. If the candidate is a chain of blocks (for example if a block arrived that filled a gap in an alternative better fork, or if we received the tip of an alternative fork that becomes better than our selection) then we could read the values for some blocks in place (1? 2?) and in the background fetch the inputs for the rest of the blocks (or maybe just fetch all in the background and validate as they become ready)
• However most of the times when syncing blocks will arrive one by one and we will evaluate the candidates each time, so we only have a window of 1 block.
• We considered splitting chain selection in two parts:
  • ChainSel1: finds the best candidate, gives it to ChainSel2 and spawns a thread to fetch the inputs.
  • ChainSel2: for each block in the candidate, wait for the inputs to arrive and validate it, to validate the whole candidate and if successful then adopt it.
• However this falls in the same problem discussed above, if blocks come one by one we won't be able to pipeline the reads.
• A second design takes advantage of the fact that we only download "plausible" candidates in BlockFetch, i.e. blocks that belong to a selection that would be better than ours. With this precondition, all blocks in the volatile DB are interesting and we would try to validate them, so as blocks arrive to the VolatileDB we could start fetching their inputs. Then we don't need to split chain selection, but we are relying on something that is uncertain if it will be preserved forever in BlockFetch.

Start with the UTxO-HD discussion in this issue, then trace Chain selection, BlockFetch, and the VolatileDB flow to understand where candidate blocks and their inputs are handled. The issue does not name files, tests, or a settled design; done would require agreeing on a pipelining approach and defining its implementation and validation scope.