cockroachdb / cockroachdb/cockroach
cli: add debug txn-record-nemesis to inject PENDING txn records via HeartbeatTxn
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
We want a controlled way to stress situations where a range holds transaction records that *claim* large sets of lock spans (e.g. fat PENDING records that later drive intent-resolution fanout when abandoned/aborted/GCed).
Today the practical injection path is crafting KV batches by hand (`debug send-kv-batch`). That works, but:
- Transaction records are internal keys; raw Puts are not a supported client write path.
- Encoding real SQL primary-index keys by hand (base64) is awkward for operators targeting an existing table’s keyspace.
- `SendKVBatch` historically rejected transactional batches (`CrossRangeTxnWrapperSender` fatal); that is being fixed so crafted `HeartbeatTxn` / `EndTxn` batches can be sent.
We need a small, purpose-built debug tool that generates synthetic txn records against table/index IDs without requiring encoded keys up front.
**Describe the solution you'd like**
### Command
```text
cockroach debug txn-record-nemesis [flags]
# or stdin when no file is given
```
Connection model matches `debug send-kv-batch`: `--url` / host+certs → Admin RPC (`SendKVBatch`). Not a SQL session.
### What it does (and does not do)
For each txn spec in the input, sequentially send one `BatchRequest` containing a single `HeartbeatTxn`, with:
- `Header.Txn` = fresh PENDING transaction
- `Txn.Key` / request key = resolved anchor key
- `Txn.LockSpans` = resolved lock keys (persisted on first create via `AsRecord()`)
**Does write:** PENDING `TransactionRecord`s that list the claimed lock spans.
**Does not write:** real intents/locks on those keys. Help text must say this explicitly. The stress target is “fat txn records → later GC/abort → IR fanout,” not “ranges full of intents.”
Records persist until abandoned-txn GC (`kv.gc.txn_cleanup_threshold`) pushes/aborts them (or until an explicit abort/commit). No concurrency/batching in v1 — sequential is fine.
### Input: protobuf-backed JSON
Define a small proto (e.g. under `pkg/cli` debug helpers) and accept JSON via `protoutil.JSONPb`, same pattern as `send-kv-batch`.
```protobuf
message TxnRecordNemesisSpec {
repeated TxnRecordNemesisTxn txns = 1;
}
message TxnRecordNemesisTxn {
// Anchor for the txn record (Txn.Key / HeartbeatTxn key).
KeyRef anchor = 1 [(gogoproto.nullable) = false];
// Claimed lock spans persisted on the PENDING record.
repeated KeyRef locks = 2 [(gogoproto.nullable) = false];
}
message KeyRef {
// Exactly one of index or key should be set.
IndexKeyRef index = 1;
bytes key = 2 [(gogoproto.casttype) =
"github.com/cockroachdb/cockroach/pkg/roachpb.Key"];
}
message IndexKeyRef {
uint32 table_id = 1;
uint32 index_id = 2;
// Optional; if unset, tool picks a large random int64.
int64 pk = 3;
}
```
Example:
```json
{
"txns": [
{
"anchor": {"index": {"tableId": 104, "indexId": 1}},
"locks": [
{"index": {"tableId": 104, "indexId": 1}},
{"index": {"tableId": 104, "indexId": 1, "pk": 999001}},
{"key": ""}
]
}
]
}
```
### Key resolution
For `IndexKeyRef`, synthesize a point key in that index’s keyspace:
```text
/Table////0
```
(family 0). This is intentionally **not** general SQL row encoding — it only needs to land in the right index keyspace. Document that assumption. Optional raw `key` is the escape hatch for arbitrary addresses.
Use the codec appropriate to the connected tenant/cluster (same rules as other debug KV tools).
### Prerequisites / related work
- `SendKVBatch` must forward transactional batches to DistSender (bypass `CrossRangeTxnWrapperSender`). Covered by local work / tests around `TestSendKVBatchInjectTxnRecord`.
- Unit/CLI test can mirror the existing HeartbeatTxn flavor in `pkg/cli/debug_send_kv_batch_test.go` (SQL table + encoded keys + QueryTxn to confirm PENDING record + LockSpans).
### v1 non-goals
- Concurrency / parallel injection
- Chunking multiple HeartbeatTxns into one `BatchRequest` (impossible: one `Header.Txn` per batch)
- `EndTxn(commit)` / real intent-resolution stress flavor (follow-up; local keys auto-GC, external keys race with async IR)
- Writing real intents
- Multi-column / non-INT primary-key encoding
- Looking up table IDs by name (operator supplies descriptor/index IDs)
### Follow-ups (optional, out of scope)
- `EndTxn(commit)` mode for IR stress (with guidance on external LockSpans)
- `--count` / per-lock expansion counts for template expansion
- Concurrency
**Describe alternatives you've considered**
1. **Raw `debug send-kv-batch` only** — works once transactional batches are allowed, but encoding keys and assembling HeartbeatTxn JSON by hand does not scale for “tons of locks on some range.”
2. **Store-level / unsafe engine writes of `TransactionRecord`** — bypasses Raft placement rules and is easy to get wrong across replicas; rejected.
3. **`EndTxn(commit)` as the primary injector** — triggers sync/async intent resolution immediately; good for IR stress, bad for leaving stable PENDING records. HeartbeatTxn is the better v1 default; EndTxn can be a later mode.
4. **Name `intentnemesis`** — oversells: we are not writing intents. Prefer `txn-record-nemesis`.
**Additional context**
Relevant internals:
- `HeartbeatTxn` creates a PENDING record from `Header.Txn` on miss (`AsRecord()` includes `LockSpans`); no intent resolution.
- `BeginTransaction` was removed (lazy txn record creation); HeartbeatTxn is the supported “create record” path.
- Local investigation / prototype path: `pkg/cli/debug_send_kv_batch_test.go` (`TestSendKVBatchInjectTxnRecord` / `heartbeat` subtest) constructs SQL primary-index keys and injects via `send-kv-batch`.
Acceptance criteria for v1:
- [ ] `cockroach debug txn-record-nemesis` reads protobuf-JSON spec from file or stdin
- [ ] Sequentially injects PENDING txn records via Admin `SendKVBatch` + `HeartbeatTxn`
- [ ] Supports `IndexKeyRef` (table/index/+optional pk) and raw `key`
- [ ] Help text clearly states: no real intents; records claim lock spans only
- [ ] CLI test covers at least one HeartbeatTxn injection + `QueryTxn` verification
Jira issue: CRDB-65876
Contributor guide
Research direction
Start with pkg/cli/debug_send_kv_batch_test.go, especially TestSendKVBatchInjectTxnRecord and its heartbeat subtest, then trace the existing send-kv-batch Admin RPC path. Implement the protobuf-JSON file or stdin flow and sequential HeartbeatTxn injection for index and raw keys. Done means the CLI test verifies a PENDING record and LockSpans with QueryTxn, and help text states that no real intents are written.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100