IntersectMBO / IntersectMBO/cardano-db-sync

only_utxo preset in bootstrap mode inserts 0 tx_out, then crashes near tip

Open
#2,150 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Haskell
Stars
318
Forks
168
PR merge metrics
No merged PRs in 30d

Description

### Summary

Running db-sync with `insert_options.preset = only_utxo` (tx_out bootstrap mode) on the default in-memory ledger backend syncs to near-tip, then the bootstrap UTxO migration inserts 0 rows, reports "done", and db-sync immediately crashes on the next block with "TxIn not found in memory".

So tx_out, tx_in and ma_tx_out end up empty, and the first spend of any pre-bootstrap UTxO cannot be resolved, which kills the db-sync thread. The failure is silent up to the crash: the migration log says it finished successfully.

There is no recovery on restart because the bootstrap is already marked finished.

The bootstrap migration log says it inserted 0 rows, then crashes:

```
Syncing with bootstrap. This won't populate tx_out until the tip of the chain.
...
Starting UTxO bootstrap migration
Inserting 0 tx_out as pages of 25000
UTxO bootstrap migration done
...
NearTip migrations were ran successfully.
Error SNErrDefault: "TxIn not found in memory: TxIn {txInIndex = 0, txInTxId = TxId {unTxId = SafeHash \"4e860b...\"}, txInRedeemerIndex = Nothing}":
runDbThread: Error SNErrDefault: "TxIn not found in memory: ...":
ChainSyncWithBlocksPtcl: AsyncCancelled
```

## Versions

- cardano-db-sync: 13.7.2.1
- cardano-node: 11.0.1
- PostgreSQL: 18.4
- Network: preview

## Config

The only thing I set under `insert_options` was the preset:

```json
"insert_options": {
"preset": "only_utxo"
}
```

No `ledger_backend` key was set in the config, so the run used the default in-memory ledger backend (LedgerBackendInMemory). This is the default and supported path, not LSM.

## What I expected vs what I got

Database fully synced to the tip (block_no 4421626, epoch 1341). Row counts:

| Table | Expected | Actual | Notes |
|---------------------|---------------|--------|-----------------------------------------|
| block | filled | 4421628| ok |
| tx | filled | 6646874| ok |
| tx_out | filled | 0 | BUG - empty, this is the core data |
| tx_in | filled | 0 | BUG - empty |
| ma_tx_out | filled | 0 | BUG - empty |
| multi_asset | filled | 612889 | ok |
| tx_metadata | empty | 0 | ok (metadata off in preset) |
| redeemer | empty | 0 | ok (plutus off in preset) |

## Bootstrap migration ran but read an empty UTxO

`extra_migrations` shows both BootstrapStarted and BootstrapFinished, so the
migration ran to completion and will not retry:

```
BootstrapStarted | The bootstrap syncing is in progress
BootstrapFinished | The bootstrap is finalised
```

So tx_out is permanently empty unless the DB is rebuilt.

## The crash that follows

Because the migration wrote 0 tx_out, the next block that spends a pre-bootstrap UTxO cannot resolve its input. In consumed/bootstrap mode, input resolution (Cardano/DbSync/Era/Universal/Insert/Grouped.hs) first queries the DB (resolveInputTxOutIdFromTxId), which fails because tx_out is empty, then falls back to in-memory resolution, which only knows outputs created in the current batch. A pre-tip UTxO is in neither, so it throws:

```
TxIn not found in memory: TxIn {...}
```

and runDbThread dies. db-sync cannot advance past the tip.

## Likely root cause (unverified, AI-assisted code analysis)

Note: everything above this point is observed behaviour (logs, row counts, config, repro) and is solid. The section below is a code-reading analysis I put together with the help of an AI assistant (Claude). It has not been confirmed with a debugger and may be wrong about the consensus / UTxO-HD internals, so please treat it as a lead rather than a conclusion.

The `only_utxo` preset uses bootstrap tx_out mode (`TxOutConsumedBootstrap`), which does not write tx_out during sync and instead imports the full UTxO once, near the tip, in `migrateBootstrapUTxO` (Cardano/DbSync/Api/Ledger.hs).

That import reads the UTxO from the in-memory ledger state via
`readCurrentStateUnsafe`, whose return type is:

```haskell
readCurrentStateUnsafe :: HasLedgerEnv -> IO (ExtLedgerState CardanoBlock EmptyMK)
```

`EmptyMK` is the UTxO-HD empty mapkind, i.e. the ledger value carries no ledger
tables - the UTxO set lives in a separate backing store. `storeUTxOFromLedger`
then projects the UTxO out of that empty state:

```haskell
getUTxO st' = unUTxO $ shelleyLedgerState st' ^. (nesEsL . esLStateL . lsUTxOStateL . utxoL)
```

On an `EmptyMK` state this map is empty, so `Map.size mp == 0`, the migration inserts 0 rows, records BootstrapFinished, and clears the bootstrap flag.

The code already guards this case for the LSM backend (Cardano/DbSync/Api.hs):

```
bootstrap-tx-out is not supported with ledger_backend=lsm. The bootstrap path
reads the full UTxO from the in-memory ledger state, which is empty under LSM.
```

The guard's assumption ("inmemory keeps the UTxO in the in-memory state") does not hold post UTxO-HD. Under V2 LedgerDB the UTxO lives in a separate tables handle for both backends (the inmemory branch uses InMem.InMemArgs), and readCurrentStateUnsafe returns EmptyMK either way.
So inmemory falls through the guard and then fails in exactly the way the lsm message describes.

Scope: only tx_out bootstrap mode (the only_utxo preset, or bootstrap set directly). Other tx_out modes (enable / consumed / prune / disable) are unaffected. lsm is blocked up front; inmemory (the default) crashes.

Note: the shutdown log line "closeLedgerEnv: closing LSM session..." appears even on the inmemory backend. That is not evidence of LSM. It is a hardcoded log string printed for any HasLedger env (Cardano/DbSync.hs), and on inmemory the close action is a no-op. The real evidence is the EmptyMK type above.

## Expected behaviour / suggested fix

Either of:

1. If bootstrap should work on inmemory: read the UTxO from the V2 LedgerDB tables handle / backing store instead of from the EmptyMK state.
2. If bootstrap should be retired with UTxO-HD: extend the existing guard to also reject bootstrap + inmemory, so it fails fast at startup with a clear message instead of crashing near the tip after a full sync.

## Steps to reproduce

1. Empty PostgreSQL database.
2. db-sync 13.7.2.1 against node 11.0.1 on preview, inmemory ledger backend.
3. Config with `insert_options: { preset: only_utxo }`.
4. Let it sync to the tip.
5. Query `SELECT count(*) FROM tx_out;` -> 0.

## Impact

The only_utxo preset produces a database with no UTxO data at all (its primary purpose), and then db-sync crashes and cannot advance past the tip. The failure is silent up to the crash: the migration log says it finished successfully, and a restart does not retry because the bootstrap is already marked finished.

## Logs

[utxo-only-preset-preview-logs.txt](https://github.com/user-attachments/files/29462396/utxo-only-preset-preview-logs.txt)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the only_utxo configuration on the in-memory backend and inspect Cardano/DbSync/Api/Ledger.hs around migrateBootstrapUTxO and storeUTxOFromLedger, then compare the existing guard in Cardano/DbSync/Api.hs. Trace input resolution in Cardano/DbSync/Era/Universal/Insert/Grouped.hs and verify the chosen behavior with the reported row counts and crash; done means bootstrap no longer records success with an empty UTxO or the unsupported configuration fails clearly before syncing.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.