sql/inspect: false-positive dangling_secondary_index_entry findings under chaos

INSPECT is emitting `dangling_secondary_index_entry` findings that do
not reproduce under direct SQL (`SELECT COUNT(*)` and `LEFT LOOKUP
JOIN … WHERE pk IS NULL`) at a fixed AOST. Observed twice on distinct
TPC-C tables on `drt-chaos-aws` in the last ~48 hours; on both
occasions direct verification showed the underlying data to be fully
consistent, i.e. the INSPECT findings were false positives.

The class of failure appears related to the row-count-mismatch false
positives previously seen on the same cluster (#165540, #168001,
#168396, #168400) but affects the index-consistency check rather than
row-count validation, and is not addressed by the mitigation in
#168400. The schema-change-during-INSPECT mitigation from #158197 is
included in the affected build but does not prevent these false
positives either.

**Observations**

Two independent INSPECT runs on the cluster produced findings that
failed to reproduce:

| Time (UTC) | Table | Index | Findings emitted | Reproduces via direct SQL? |
|---|---|---|---|---|
| 2026-06-29 15:24Z | `cct_tpcc.customer` | `customer_idx` | 4,973 `dangling_secondary_index_entry` | No — INSPECT reruns at 22:34Z and 22:35Z on the same day returned `issuesFound=false`. |
| 2026-07-01 00:24Z | `cct_tpcc.public.order` | `order_idx` | 1,807 `dangling_secondary_index_entry` (+ some `missing_secondary_index_entry` on the same table at the same AOST) | No — see verification below. |

Additionally, two IMPORT-triggered INSPECT jobs (`1188855042320465921`
on `customer`, `1188854244957224961` on `order`) failed with "INSPECT
found inconsistencies" at ~04:30Z on 2026-07-01, causing the
associated IMPORTs to be reverted. These likely fall under the
#165540 / #168400 family but appear to affect a check that is not
disabled by `bulkio.import.row_count_validation.mode = off`.

**Verification (order.order_idx, 2026-07-01)**

Ran in a single transaction against `drt-chaos-aws` gateway at a
fixed AOST snapshot (~30h after the INSPECT finding):

```sql
BEGIN AS OF SYSTEM TIME '-30s';
SELECT COUNT(*) FROM cct_tpcc.public.order@order_pkey;
-- 739,875,560
SELECT COUNT(*) FROM cct_tpcc.public.order@order_idx;
-- 739,875,560
SELECT idx.o_w_id, idx.o_d_id, idx.o_id
FROM cct_tpcc.public.order@order_idx AS idx
LEFT LOOKUP JOIN cct_tpcc.public.order@order_pkey AS pk
ON pk.o_w_id = idx.o_w_id AND pk.o_d_id = idx.o_d_id AND pk.o_id = idx.o_id
WHERE pk.o_w_id IS NULL
LIMIT 100;
-- 0 rows
COMMIT;
```

PK and secondary index counts agree exactly and no dangling entries
exist. The 30 specific `(o_w_id, o_d_id, o_id)` PKs that INSPECT
sampled and logged as dangling at 00:24Z were all present on both
sides at query time.

Note: an earlier attempt to run the two COUNTs as separate
statements (`SELECT ... AS OF SYSTEM TIME '-30s'` outside a
transaction) produced a spurious 35,552-row delta. Each statement
picks its own AOST, and with ~185s of scan time between them, the
TPC-C new-order workload had inserted enough rows to fake a
divergence. Using `BEGIN AS OF SYSTEM TIME` binds all statements in
the txn to a single HLC and eliminates this artifact.

**Why the workload cannot explain \"self-healing\"**

- TPC-C `customer` UPDATEs (payment, delivery) touch `c_balance`,
`c_ytd_payment`, `c_payment_cnt`, `c_data`, `c_delivery_cnt`.
`customer_idx` is on `(c_w_id, c_d_id, c_last, c_first)` — none of
those columns are updated by workload after load. TPC-C does not
DELETE from `customer`. There is no path for real dangling
entries at 15:24Z to disappear by 22:34Z through workload
activity.
- TPC-C `order` is INSERT-only via new-order; `o_id` is monotonic
per district and is never re-used. Real dangling entries at
00:24Z would still be present at query time.

Combined with the direct-SQL verification, this rules out
transient real corruption that \"self-healed\" and leaves an
INSPECT-internal false-positive class as the explanation.

**Environment**

- CockroachDB version: `v26.3.0-alpha.1-dev`
- Build SHA: `b7dbd46c6f553d08b626ad09975e59fd27d640d8` (built 2026-06-09)
- Cluster: `drt-chaos-aws` (6-node AWS us-east-2, EBS)
- Concurrent workloads: TPC-C (12,000 warehouses), KV, gitload

**Related issues**

- #165540, #168001, #168396 — row_count_mismatch false positives on same cluster
- #168400 — mitigation that disabled `bulkio.import.row_count_validation.mode` by default

Epic CRDB-65516