Improve storage efficiency of types.Datum

## Enhancement
types.Datum currently takes 72 bytes, so more than single cache line (normally 64 bytes).
There is no need to store a string for the collation, it can be done as an enum/int16 instead, so it is possible to shrink it down to 56 bytes instead, so it fits in a single cache line.

Also any part that stores many Datum's will benefit from lower memory needs, like keeping all histograms in memory during merging etc.

---

**Status (2026-04-24):** the original 56 B goal has been reached and exceeded — the full A+B+pre-C+C stack (PRs #67977, #67978, #68007, #68011) brings `types.Datum` to **48 B**, a **−33%** reduction, with `make([]Datum, 100)` allocations going from 8192 B → 4864 B (**−40.6%**). Details in the Milestones below.

---

## Plan

The work is split into small, independently reviewable PRs. Each milestone is
either landed, in review, or explicitly punted for later. Measurements are on
linux/amd64, Intel i7-13700, `benchstat -count=5`.

### Current `types.Datum` layout (master, 72 B)

```
offset size field
0 1 k byte
1 1 (padding)
2 2 decimal uint16 // frac; MySQL cap 30
4 4 length uint32 // flen; only ever read for DECIMAL precision
8 8 i int64 // int/uint/float bits, Duration ticks, Enum/Set value
16 16 collation string // one of ~7 fixed MySQL collation names per Datum
32 24 b []byte
56 16 x any // *MyDecimal, Time (boxed!), nothing else
```

Key waste:
1. `collation string` — 16 B per Datum to name one of a fixed small set.
2. `decimal uint16` — 2 B where max value is 30.
3. `x any` holding `Time` — boxing a non-pointer value heap-allocates 8 B on every `SetMysqlTime`.
4. `length uint32` — set in many places, but only ever *read* for `KindMysqlDecimal` precision (3 non-test sites across the whole repo).

### Milestones

#### ✅ Benchmark baseline (landed in PR #67977)
- `b.ReportAllocs()` on existing Datum-adjacent benches in `pkg/types`, `pkg/tablecodec`, `pkg/util/codec` so `benchstat` can diff `B/op` and `allocs/op`.
- New `BenchmarkDatumCopy`, `BenchmarkDatumSetMysqlTime`, `BenchmarkCompareDatumCollation` to expose struct-size, Time-boxing, and named-collator-compare costs respectively.

#### ✅ Option A — Shrink 72 → 56 B (PR #67977)
- `collation string` (16 B) → `collationID uint16` (2 B) carrying MySQL collation id; 0 = unset (matches legacy empty-string sentinel).
- `decimal uint16` → `uint8`. Frees 2-byte hole that the new `collationID` absorbs; first word stays 8 B.
- `Collation() string` / `SetCollation(string)` preserved as facades via the `charset` package; new `CollationID() uint16` / `SetCollationID(uint16)` for hot paths.
- `MarshalJSON`/`UnmarshalJSON` keep the wire `"collation"` field as a name string, so older JSON blobs round-trip.
- Hot-path callers (codec encode, ranger, planner/cardinality, table/mutation_checker, executor/foreign_key, plus internal Datum compare/key paths) migrated to `collate.GetCollatorByID(int(d.CollationID()))` to avoid a second name→ID lookup.
- `collate.GetCollatorByID(0)` now falls through silently (matches `GetCollator("")`).
- `Datum.MemUsage` drops the `len(collation)` term; collation is fully inline in `EmptyDatumSize` now.

**Results:**
- `unsafe.Sizeof(types.Datum)`: **72 → 56 B (−22%)**
- `BenchmarkDecodeWithSize` (codec): **8 KiB → 6 KiB B/op (−25%), 2435 → 1862 ns/op (−24%)**
- `BenchmarkDecodeWithOutSize` (codec): **20.88 KiB → 15.83 KiB (−24%)**
- `BenchmarkCompareDatum` (types): **34.06 → 32.12 ns/op (−5.7%)**
- `BenchmarkEncodeValue` (tablecodec): **135 → 122 ns/op (−10%)**
- No allocation regressions

#### ✅ Option B — Pack Time into `d.i` (PR #67978)
Independent of Option A, branched directly off master.

- `Time = struct{CoreTime}` where `CoreTime = uint64` → fits in `d.i` exactly.
- Storing it via `d.x any` previously allocated 8 B per `SetMysqlTime` (Go boxes non-pointer interface payloads).
- `SetMysqlDuration` already packed into `d.i`; this mirrors the pattern for Time.
- Removes the `KindMysqlTime` branch in `Datum.Copy` (bitwise `*dst = *d` is now a complete value copy).

**Results:**
- `BenchmarkDatumSetMysqlTime`: **7.2 → 0.1 ns/op (−98.6%), 1 alloc → 0 allocs**
- `BenchmarkDatumCopy`: **46 → 37 ns/op (−20%), 64 → 53 B (−17%), 3 → 2 allocs (−33%)**
- Struct size unchanged (72 B standalone; 56 B when combined with Option A).

#### ✅ Remove KindInterface escape hatch (PR #68007)
Prerequisite for Option C. `types.Datum.SetInterface` / `GetInterface` and the `KindInterface` kind (value 14) existed as an escape hatch for Go values that `SetValue`'s type switch doesn't recognize, stored in `d.x` (`any`). An audit found:

- **Production** had exactly one caller: a parser grammar bug in `TABLESAMPLE` that wrapped an already-`ExprNode` value via `ast.NewValueExpr`, redundantly putting a whole expression tree into a `Datum`. Fixed in `parser.y`: pass `$4` and `$7`/`$5` directly (both are declared `%type `). Regenerated `parser.go` via `make parser`.
- Every other `NewValueExpr(…)` site in the grammar (66 of them) passes a primitive lexer value that goes through the native `SetValue` switch.
- Test-only idioms using `NewDatum(errors.New(...))` to trigger error paths, plus `NewDatum(someDatum)` pass-through helpers, migrated across ~30 test sites.

After this PR: `d.x` is guaranteed to only ever hold a `*MyDecimal` (or nil), unblocking Option C. The constant is renamed to `KindInterfaceDeprecated` (value 14 reserved) so external consumers that enumerate kinds get a compile-time nudge.

#### ✅ Option C — Replace `x any` with a typed `*MyDecimal` (PR #68011)
With the escape hatch gone, swap the 16-byte `any` header for a plain `decPtr *MyDecimal` (8 bytes, GC-safe, type-safe, no `unsafe`). Struct drops to **48 B**.

Memory / access comparison (justifies the chosen shape over alternatives):

| Field shape | struct bytes | heap bytes | total | access |
| --- | ---: | ---: | ---: | --- |
| `x any` (master) | 16 | 40 (MyDecimal) | **56** | 1 type assertion |
| `x *MyDecimal` (landed) | 8 | 40 | **48** | 1 pointer deref |
| `x *any` | 8 | 16 (iface) + 40 | **64** | 2 derefs + iface unbox |
| `x unsafe.Pointer` | 8 | 40 | **48** | 1 deref + type cast |

`*MyDecimal` dominates: same 8-byte struct footprint as `unsafe.Pointer` or `*any`, without the `unsafe` import and without the extra heap-allocated iface header per Set.

**Results** (A+B+pre-C → A+B+pre-C+C, benchstat n=3):
- `unsafe.Sizeof(types.Datum)`: **56 → 48 B (−14%)**
- `BenchmarkDecodeWithSize` (codec, `make([]Datum, 100)`): **6144 → 4864 B/op (−20.8%)**
- `BenchmarkDecodeWithOutSize` (codec): **15.83 KiB → 15.33 KiB (−3.2%)**
- Codec B/op geomean: **−12.4%**

**Cumulative from master**: `BenchmarkDecodeWithSize` goes 8192 → 6144 (A) → 4864 (A+B+pre-C+C) = **−40.6% B/op**. `unsafe.Sizeof(types.Datum)` goes 72 → 48 = **−33%**.

Bonus: while implementing C, caught and fixed a latent Option A bug — `GetMysqlDuration` needed `int(int8(d.decimal))` sign-extension to preserve the `Fsp = -1` ("unspecified") sentinel once `decimal` narrowed from `uint16` to `uint8`. `TestMemJsonObjectagg` surfaced it as a `slice[0:255]` panic in `Duration.formatFrac`.

### Future candidates (not started)

##### Alternative: encode `MyDecimal` into `b []byte` (target 40 B)
Instead of a separate decimal pointer, the encoded decimal bytes could live in `d.b` and the `x`/`decPtr` slot disappears entirely:

```
k(1) decimal(1) collationID(2) length(4) i(8) b(24) = 40 B
```

Pros: 8 more bytes saved; fewer kind-specific fields; unifies "variable-length data" into one slot.

Cons, all real:
- `GetMysqlDecimal` goes from a ~1 ns type assertion to a full `DecodeDecimal` (~29 ns on master, 1 alloc for the fresh `*MyDecimal`). `SetMysqlDecimal` goes from a pointer store to a `WriteBin` serialize.
- Loses pointer aliasing: today a computed `*MyDecimal` can be shared across Datums cheaply; encode-to-`b` forces a copy per Set.
- Overloads `d.b` (currently "string / bytes / enum-name / set-name / JSON / binary-literal payload") with a new meaning.

This is appealing only for workloads that **store** many decimals without accessing them often (e.g., in-memory histograms). For OLTP paths that touch each Datum at least once, the CPU cost of per-access decode likely outweighs the 8 B struct saving. Keep as a targeted follow-up gated on profiling evidence, not as the default Option C.

#### ▢ Dead-write audit for `Datum.length`
Non-test call graph shows `Length()` is read in exactly three places, all for DECIMAL precision (`util/codec/codec.go:121,170`, `util/rowcodec/encoder.go:214`). For strings/bytes/enum/set/JSON, `length` is set but never read. These are dead writes that also cause false inequalities in `Datum.Equals` (plan-cache constant equality, etc.). A small targeted cleanup PR can stop the non-decimal writes and tighten the `Equals` path without changing on-disk format.

#### ▢ Fold decimal precision into `d.i` → drop `length` → 44/40 B
`d.i` is unused when `k == KindMysqlDecimal` (the pointer goes in `d.x`). Storing precision/frac in `d.i` for decimals and removing the separate `length` field would avoid changing the on-wire decimal encoding (which today uses declared precision byte) while freeing 4 more bytes of struct. Together with Option C this could reach **40 B**. Requires careful thought about `Datum.Equals` semantics for non-decimal Datums.

### End-to-end validation (sysbench, 3-way same-day comparison)

> ⚠️ **Hardware / environment caveat — read first.**
>
> All sysbench numbers on this page were produced on a single-user desktop workstation (Intel Core i7-13700, Linux 5.14) running a colocated TiUP playground (PD + TiKV + TiDB on the same host), with no isolation from other running processes, no CPU pinning, no thermal controls, and only the TiDB binary varying between runs. This is explicitly **NOT a controlled benchmark environment** — it is a developer-box smoke test, suitable for spotting gross regressions / confirming a micro-benchmark signal survives to the SQL layer, not for publishing performance claims.
>
> Every number below should be repeated on server-spec hardware in an isolated, controlled setup (dedicated machines for TiDB / TiKV / PD / sysbench client, CPU frequency/governor pinned, thermal headroom verified, power state locked, longer runs, higher n) before being used to justify decisions beyond "does the change directionally move a signal". Treat the results as indicative, not definitive.
>
> The original 2-way measurement taken on 2026-04-22 (reported earlier in this issue) showed a +5.7% QPS / −5.2% p95 win on `oltp_point_select` for A+B; running the same binaries 2 days later put A+B essentially flat against master. Almost all of that previous "win" was cross-day hardware/thermal drift, not the code change. The 2-way cross-day numbers are retired. Only the same-day 3-way numbers below are load-bearing — and even those carry the caveat above.

Setup (same across all three binaries): sysbench 1.0.20, threads=64, 8 tables × 100 000 rows, 1 × 60 s warmup + 3 × 120 s measured per workload. PD/TiKV pinned to `v9.0.0-beta.2.pre-nightly` (TiDB master expects RPCs newer than v8.5.6's `QueryRegion`). Binaries compiled with the same Go toolchain and `-trimpath`; only TiDB source differs.

Per-run QPS:

| Workload | Run | base QPS | A+B QPS | full (A+B+pre-C+C) QPS |
| --- | ---: | ---: | ---: | ---: |
| `oltp_point_select` | 1 | 70 222.92 | 69 881.91 | 70 121.94 |
| `oltp_point_select` | 2 | 70 749.19 | 70 280.36 | 72 540.71 |
| `oltp_point_select` | 3 | 70 807.99 | 70 684.00 | 71 357.34 |
| `oltp_read_only` | 1 | 45 127.57 | 45 172.35 | 46 244.77 |
| `oltp_read_only` | 2 | 45 069.65 | 45 072.93 | 46 067.52 |
| `oltp_read_only` | 3 | 45 056.06 | 45 089.91 | 45 656.51 |

Per-run p95 (ms):

| Workload | Run | base p95 | A+B p95 | full p95 |
| --- | ---: | ---: | ---: | ---: |
| `oltp_point_select` | 1 | 1.55 | 1.55 | 1.52 |
| `oltp_point_select` | 2 | 1.52 | 1.52 | 1.50 |
| `oltp_point_select` | 3 | 1.52 | 1.52 | 1.50 |
| `oltp_read_only` | 1 | 28.67 | 28.67 | 27.66 |
| `oltp_read_only` | 2 | 28.67 | 28.67 | 28.16 |
| `oltp_read_only` | 3 | 28.67 | 28.67 | 28.16 |

Mean + range-overlap check:

| Workload | Metric | base → A+B | A+B → full | base → full | Overlap (base→full)? |
| --- | --- | ---: | ---: | ---: | --- |
| `oltp_point_select` | QPS | 70 593 → 70 282 (−0.44%) | 70 282 → 71 340 (+1.51%) | 70 593 → 71 340 (+1.06%) | Yes (min(full) 70 122 < max(base) 70 808) |
| `oltp_point_select` | p95 ms | 1.53 → 1.53 | 1.53 → 1.51 (−1.31%) | 1.53 → 1.51 (−1.31%) | Yes (max(full) 1.52 = min(base) 1.52) |
| `oltp_read_only` | QPS | 45 084 → 45 112 (+0.06%) | 45 112 → 45 990 (+1.95%) | 45 084 → **45 990 (+2.01%)** | **No — min(full) 45 657 > max(base) 45 128** |
| `oltp_read_only` | p95 ms | 28.67 → 28.67 | 28.67 → 27.99 (−2.37%) | 28.67 → **27.99 (−2.37%)** | **No — max(full) 28.16 < min(base) 28.67** |

**Takeaways (indicative, see caveat above):**
- **A+B alone is a wash at this scale.** Point-select and read-only both fall within noise vs master (overlapping ranges, sub-1% means). The earlier cross-day +5.7% result was drift, not signal.
- **The full stack (A+B+pre-C+C) lands a directional win on `oltp_read_only`:** +2.01% QPS, −2.37% p95, both non-overlapping ranges in this same-day set. Matches expectation — read-only exercises `[]Datum` allocations across range scans and joins, so the 72 → 48 B per-Datum reduction has more surface area than a single-row point-select lookup does.
- **Point-select with the full stack** trends positive (+1.06% QPS / −1.31% p95) but ranges overlap. Not a claimable win at n=3.
- The delta between A+B and full is where the measurable movement sits: pre-C (#68007) and C (#68011) together moved `oltp_read_only` out of noise on this box. A+B's `BenchmarkDecodeWithSize` micro-benchmark win (8 KiB → 6 KiB per 100-Datum slice) didn't visibly materialize at the SQL layer at this concurrency.

**Decision:** the full A+B+pre-C+C stack shows a directional end-to-end improvement on this setup, with a non-overlapping range on `oltp_read_only`. Before this drives any further decisions, the whole protocol should be repeated on a controlled server-spec rig — the hardware caveat at the top of this section is the binding constraint on how much weight these numbers can carry.

### Originally planned protocol (if/when a bigger rig is available)

- `oltp_point_select` — narrowest kernel; most sensitive to per-Datum cost.
- `oltp_read_only` — realistic mixed reads.
- Optionally `oltp_update_index` — more Time/Decimal churn.

3 warm runs, 5 measured runs of 120 s each at threads=64 and threads=256; compare QPS, p95, CPU-per-txn against master.

### Non-goals / out of scope

- No change to on-disk or network encoding (Datum is an in-memory type; JSON marshal stays name-based).
- No behavior change visible to SQL users.
- No deprecations of public `types` APIs — `Collation() string`, `SetCollation(string)`, `Frac()/SetFrac(int)` keep their signatures.