Checksum error during concurrent DOLT_COMMIT: writeCommitParentClosure: ReadManyValues

# Dolt Bug Report: Checksum Error During Concurrent Commits

## Summary

Multiple concurrent connections performing `DOLT_COMMIT` operations on the same database result in journal corruption and `checksum error` failures. This occurs in a production indexer workload where two services simultaneously write to the same Dolt database.

## Environment

- **Dolt Version**: 1.79.2
- **Deployment**: Docker container (`dolt sql-server`)
- **OS**: Linux (Ubuntu)
- **Storage**: Docker volume on ext4 filesystem
- **Disk**: 4TB, 3% used (no disk space issues)

## Reproduction Scenario

### Setup
- Two blockchain indexers (for different chains: Sepolia and BSC Testnet) share the same Dolt database
- Each indexer performs frequent operations:
- `DOLT_COMMIT` after processing each block
- `DOLT_TAG` to create block hash tags
- `DOLT_CHECKOUT` for branch management
- Auto GC is enabled (default settings)

### Workload Pattern
- ~2 concurrent connections actively writing
- Each connection performs commits every few seconds
- Tags created/deleted frequently (idempotent tag creation pattern)

## Error Details

### Primary Error
```
error="writeCommitParentClosure: ReadManyValues: checksum error"
```

### Secondary Error (on other connections)
```
error="checksum error"
```

### FSCK Output
```
WARNING: Chunk journal is corrupted and some data may be lost.
Run `dolt fsck --revive-journal-with-data-loss` to attempt to recover the journal by
discarding invalid data blocks.
```

## Timeline

| Time (UTC) | Event |
|------------|-------|
| 2026-01-10 16:06:28 | Database `dev_e9a7c7681cd88bfc` created |
| 2026-01-10 16:07:20 | Two indexers start writing (connections 163013, 163014) |
| 2026-01-10 16:11:39 | First Auto GC completes successfully (13.4s) |
| 2026-01-10 16:15:32 - 18:18:47 | Auto GC runs successfully every ~4-5 minutes |
| 2026-01-10 18:18:47 | Last successful Auto GC (2.1s) |
| **2026-01-10 18:22:34** | Connection 165105 established |
| **2026-01-10 18:22:41** | Connection 165105: `writeCommitParentClosure: ReadManyValues: checksum error` |
| 2026-01-10 18:22:41 | Connection 165105 closed |
| **2026-01-10 18:24:04** | Connection 164946: `checksum error` |

## Log Excerpts

### Connection 165105 (First Checksum Error)
```
time="2026-01-10T18:22:34Z" level=info msg=NewConnection DisableClientMultiStatements=false connectionID=165105
time="2026-01-10T18:22:41Z" level=warning msg="error running query" connectTime="2026-01-10 18:22:41.024533664 +0000 UTC" connectionDb=dev_e9a7c7681cd88bfc connectionID=165105 error="writeCommitParentClosure: ReadManyValues: checksum error" queryTime="2026-01-10 18:22:41.424284026 +0000 UTC"
time="2026-01-10T18:22:41Z" level=info msg=ConnectionClosed connectionID=165105
```

### Connection 164946 (Second Checksum Error)
```
time="2026-01-10T18:24:04Z" level=warning msg="error running query" connectTime="2026-01-10 18:24:03.187784724 +0000 UTC" connectionDb=dev_e9a7c7681cd88bfc connectionID=164946 error="checksum error"
```

### Auto GC Before Error
```
time="2026-01-10T18:18:47Z" level=info msg="sqle/auto_gc: Successfully completed auto GC of database dev_e9a7c7681cd88bfc in 2.125418829s"
```

## Affected Databases

Two databases showed the same corruption pattern:
- `dev_e9a7c7681cd88bfc`
- `dev_c21f1ed3e870744d`

Both had similar journal file sizes (~123MB) and showed identical `fsck` warnings.

## Manifest File Content

```
5:__DOLT__:ljjksvr14aa4mkiesp6p4894e7km060h:3g55bqgr3o05l3m0sp9pfk6svik12j12:gvt31air2s95qofvbn0ohtped6spchuh:e04uqi527nsj8clqtbkf1iha25csgsuv:14485:vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv:11
```

## Analysis

### Observations

1. **Concurrent Commits**: Two indexers were performing `DOLT_COMMIT` operations simultaneously on the same database
2. **Error Location**: The error occurred in `writeCommitParentClosure`, which computes the transitive closure of parent commits during commit
3. **Multiple Affected Connections**: Two different connections (165105 and 164946) encountered checksum errors within 1.5 minutes
4. **GC Not Directly Involved**: The last successful GC was ~4 minutes before the first error, suggesting this isn't a GC race condition

### Hypothesis

The corruption appears to be caused by a race condition when multiple connections perform concurrent commits:

1. Connection A starts `DOLT_COMMIT`, begins writing to journal
2. Connection B starts `DOLT_COMMIT`, also begins writing to journal
3. Both connections write chunks to the journal file
4. Race condition causes chunk data to be written incorrectly or partially
5. Later reads find chunks with invalid checksums

## Application Code Pattern

Our indexer uses this pattern for commits:

```rust
// Simplified version of our commit pattern
pub async fn dolt_commit(conn: &mut C, message: &str) -> Result {
conn.exec_drop("CALL DOLT_ADD('-A')", ()).await?;
conn.exec_drop("CALL DOLT_COMMIT('-m', ?, '--allow-empty')", (message,)).await?;
dolt_head(conn).await
}

pub async fn dolt_tag(conn: &mut C, tag_name: &str) -> Result<()> {
// Try to delete existing tag first (idempotent)
let _ = conn.exec_drop("CALL DOLT_TAG('-d', ?)", (tag_name,)).await;
conn.exec_drop("CALL DOLT_TAG(?)", (tag_name,)).await?;
Ok(())
}
```

Each indexer:
1. Processes a block
2. Calls `DOLT_COMMIT` to commit changes
3. Calls `DOLT_TAG` to create a tag for the block hash

## Questions

1. Is concurrent `DOLT_COMMIT` from multiple connections supported and safe?
2. Should we serialize commits at the application level?
3. Is there a known issue with `writeCommitParentClosure` under concurrent load?
4. Are there any configuration options to mitigate this?

## Workaround

Currently considering:
- Using separate databases for each indexer
- Implementing application-level locking around commits
- Upgrading to a newer Dolt version (if this is fixed)

## Additional Information

- The "tag not found" errors in logs are expected (our code tries to delete tags before creating them)
- No OOM or system-level errors in `journalctl` around the time of corruption
- Docker container has not restarted (running since Dec 31, 2025)
- No disk I/O errors detected

## Attachments

If needed, we can provide:
- Full docker logs around the time of the error
- Journal file hexdump
- Complete indexer source code