UTxO-HD 2.0

The current design of UTxO-HD caused its concepts to leak everywhere. All downstream code needs to know about the LedgerTables and the abstract layer has to manage those.

The first question to ask would be "Why was it this way?". LedgerDB V1 kept a DbChangelog with diffs that would extend further than the immutable tip, and therefore would (in principle) have no associated LedgerState. This prompted the existence of "Diffs" and "Tables" as a concept separate from a LedgerState, needing then a definition for the HardForkBlock and using it everywhere in the codebase.

On LedgerDB V2, tables which are not ephemeral are now always `Values`. It just so happens that we split the steps for applying a block in tick > getKeys > read > apply > diff > applyDiff.

We are following this diagram, moving between the HF layer and the SingleEra layer:

```
| HFTxIn Map (TxIn (HFB xs)) (HFTxOut xs) Map HFTxIn (Delta (HFTxOut era))
| Keys HFB Values HFB Diffs HFB
HF Layer | block neededKeys -(read)-> UTxOs Diffs -(write)-> done
| \ / \ /
------------------- |----------------------------------------------------------------------------------------------------------------------------------------
| \ / \ /
Current Block Layer | eraBlock - neededKeys eraUTxOs -(apply)-> eraUTxOs' -(diff) -> utxoDiffs
| Keys B Values B Values B Diffs B
| SL.TxIn
```

But there is an important fact that we can leverage:

> PROP: Tables backend when the tip is at era `X` is isomorphic to the backend if it was in any of the eras in `{X, X+1, ...}`

This means that in the LSM database, what is on the disk can be interpreted into any era as long as it doesn't contain values that come from a later era. This is guarded simply by construction.

Leveraging this, we could do the following process instead:

```
HF Layer | block
| \ PROP
--------------------| ---------------------------------------------------------------------------------------------------------------------------------------
| \
Current Block Layer | ( eraBlock - neededKeys -(read)-> eraUTxOs -(apply)-> eraUTxOs' -(diff) -> utxoDiffs -(write)-> done)
| Keys B Values B Values B Diffs B
| TxIn TxIn TxOut era Map SL.TxIn (Delta (SL.TxOut era))
```

Therefore eliminating the need for HF level machinery and even the machinery at the abstract layer.

----------------------------------------------------------------------------

## How would this look like in the code

We could make this change in the Shelley instance of a LedgerState:
```diff
-data instance LedgerState (ShelleyBlock proto era) mk = ShelleyLedgerState
+data instance LedgerState (ShelleyBlock proto era) = ShelleyLedgerState
{ shelleyLedgerTip :: !(WithOrigin (ShelleyTip proto era))
, shelleyLedgerState :: !(SL.NewEpochState era) -- Ledger concept of a LedgerState
, shelleyLedgerTransition :: !ShelleyTransition
- , shelleyLedgerTables :: !(LedgerTables blk mk)
+ , shelleyLedgerTablesHandle :: !(LedgerTablesHandle m blk)
, shelleyLedgerLatestPerasCertRound :: !(StrictMaybe PerasRoundNo)
}
deriving Generic
```

(Possibly hiding the `m` in some existential)

And then the applyHelper function _for Shelley_ could get the block key sets, read the values, inject them into the NewEpochState, compute the diff, and create a new handle for the returned LedgerState with the diffs applied. Something like:

```haskell
applyHelper f cfg blk stBefore = do
keys <- getBlockkeySets blk
utxos <- readUTxOs shelleyLedgerTablesHandle stBefore

let TickedShelleyLedgerState
{ tickedShelleyLedgerTransition
, tickedShelleyLedgerState
} = stBefore

ledgerResult <-
f
globals
(tickedShelleyLedgerState `stow` utxos)
( let b = shelleyBlockRaw blk
h' = mkHeaderView (SL.blockHeader b)
in SL.Block h' (SL.blockBody b)
)

let track ::
Map SL.TxIn (TxOut era) ->
Map SL.TxIn (Delta (TxOut era))
track = calculateDifference utxos

newHandle <- duplicateWithDiffs shelleyLedgerTablesHandle (track newUtxos)
return $
ledgerResult <&> \newNewEpochState ->
ShelleyLedgerState
{ ...
, shelleyLedgerTablesHandle = newHandle
, ...
}
```

Further simplifications would come with this:
- No need for `mk` anywhere in the abstract layer
- The LedgerDB will again be a sequence of LedgerStates only
- Delete all the HF level machinery for TxIns, TxOuts, HF Queries, Traversing tables etc etc.
- No need to define any of these for any new tables that belong only to shelley.

Even if we were to do this, the handles of the InMemory backend could be also very simplified to just have a pointer to the `UTxO era`, and we could even inject the whole UTxO set into the Ledger in that case (by having `read = \_keys -> pure wholeTables`) and have the ledger translate the UTxO set for us instead of needing `CanUpgradeLedgerTables`.

Start by tracing the Shelley LedgerState instance and its applyHelper flow, then inspect the LedgerTablesHandle, LedgerDB, and LSM database machinery described in the issue. The work is done when the proposed Shelley-owned table handling is implemented and the HF-level machinery and abstract-layer mk dependencies identified here are no longer needed.

Written by the indexing model from the issue text.