storage: per-KV checksums across storage engine boundary

## Motivation

The storage engine is the source of truth for a node. Raft and KV rely on the integrity of the storage engine's data, and breaches of its integrity can result in false query results, data loss and a myriad of other issues depending on where corruption occurs. Today, integrity of data is largely insulated from physical corruption of persisted data (eg, [bit rot](https://en.wikipedia.org/wiki/Data_degradation)) by sstable block checksums. Every block contains a checksum computed over the block's data, and this checksum is validated whenever the block is loaded into the block cache. This protection has preserved data integrity for customers many ([[1]](https://github.com/cockroachlabs/support/issues?q=is%3Aissue+block+checksum)) times.

But the persistent storage media is not the only potential avenue for corruption. We've observed instances of corruption ([[1](https://github.com/cockroachlabs/support/issues/2063)], [[2](https://github.com/cockroachlabs/support/issues/2583)], [[3](https://github.com/cockroachlabs/support/issues/665)]) that strongly suggest corruption of data held in-memory. These can be the result of memory safety bugs within Cockroach itself, bugs in virtualization, the kernel, etc, or RAM hardware issues. It's proven especially difficult to root cause issues like this, often resulting in an unsatisfying explanation to a customer. The immense surface area over which the corruption might've occurred makes it intractable to manually comb for related memory safety bugs that could cause any individual corruption. Once in-memory corruption is persisted durably, the entire node must be replaced and its data up-replicated.

With disaggregated storage, durability of Cockroach's data is dependent on the blob storage provider's durability because 1 blob may be shared across all replicas. If a key is corrupted in-memory during a compaction while writing out the blob, the single instance of memory corruption can cause replicated data loss.

Today, [`roachpb.Value`](https://github.com/cockroachdb/cockroach/blob/master/pkg/roachpb/data.proto#L84-L86)s contain a 4-byte checksum computed over the key and value. However, `roachpb.Value`s are not used everywhere, and the checksum is explicitly not used for MVCC tombstones which assign semantic significance to the empty value. They also don't apply to invisible Pebble internal keys, like LSM tombstones. These checksums are infrequently verified, at times allowing corruption to be replicated [[1](https://github.com/cockroachdb/cockroach/issues/110572)].

Over its lifetime, an individual KV pair may be read from durable storage, copied and reencoded, and written again to durable storage many times by the storage engine. Every instance is an opportunity to introduce memory corruption and persist it.

## Per-KV checksumming

We should consider implementing per-kv checking summing within Pebble. Eventually this may deprecate and replace the `roachpb.Value` checksum, through integrating Pebble's scheme across the API boundary between Pebble and Cockroach. This proposal draws heavily from recent work in RocksDB [[1](https://rocksdb.org/blog/2022/07/18/per-key-value-checksum.html)].

Let `Checksum(,)` stand in for some to-be-determined hashing algorithm.

As user keys, values, versioned keys, Pebble internal keys, etc move and copy across the database, they're frequently rearranged and reorganized. Checksumming is at its most effective and performant when performed end-to-end, but this reorganization of data is an obstacle. RocksDB's per-KV checksumming works around this through computing per-datum checksums, XOR-ing the constituent checksums to produce a single checksum over a KV. This approach eases end-to-end integration of checksums with little additional overhead.

We define a series of checksum component types, each with a unique seed.
* Ukey — In storage engine terminology, this is a key "prefix". A user key always includes the trailing 0x00-byte terminator.
* Timestamp — In storage engine terminology, this is a key "suffix". It's computed over the engine key representation of the timestamp.
* Trailer — Computed over the 8-byte pebble InternalKey trailer containing the sequence number and key kind of an internal key. [TODO(Jackson): There's a problem here with the obsolete bit, which doesn't leave the blockIter. Do we just exclude the obsolete bit from the checksum?]
* Value — Computed over the value.

Checksums accompanying data are formed by XOR-ing the checksums of each individual datum that's available. For example, the versioned MVCC key `"apple@5"` has a user key `"apple"` and a timestamp `@5`. The checksum for the entire versioned MVCC Key is `Checksum(Ukey, "apple") ⊕ Checksum(Timestamp, @5)`.

## Persisting checksums
### ssblocks
We may introduce a new sstable format that supports a flag indicating whether or not the contained keys are checksummed. If they are, every KV-pair is trailed by a fixed-length checksum formed by XOR-ing all the individual components. If a value is stored out-of-band, the in-band checksum does not cover the value, and the out-of-band value is trailed by its own checksum. These checksums may be ferried alongside the data throughout the storage engine. During a compaction, they may be persisted verbatim.

Once finished writing an sstable, we may read back the entire sstable and validate all of the per-KV checksums (and implicitly, the ssblock checksums). If any of the checksums fail, the flush/compaction/ingest is failed, avoiding persisting the corruption.

At read time we can be assured we read the correct KVs if block checksum validation passes, since the per-KV checksums are covered by the ssblock checksums and were validated alongside the ssblock checksums at the time of writing. This still leaves open the block in-memory within the block cache vulnerable.
### memtable
TODO
### batch
TODO

## Challenges

The largest challenges are around API boundaries and performance. Many codepaths cannot tolerate ferrying an additional 8-byte or 4-byte digest by copying (eg, `base.LazyValue`). The checksum can be contained at the beginning or end of byte slices (eg, like the `roachpb.Value` does today), but we'll need well-typed interfaces to disambiguate the original values from checksum-suffixed values.

Jira issue: CRDB-31709