cockroachdb / cockroachdb/cockroach

cli: add debug txn-record-nemesis to inject PENDING txn records via HeartbeatTxn

Open
#172,629 1 comment 0 reactions 0 assignees View on GitHub
A-kv-transactions C-enhancement E-starter O-agent T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We want a controlled way to stress situations where a range holds transaction records that *claim* large sets of lock spans (e.g. fat PENDING records that later drive intent-resolution fanout when abandoned/aborted/GCed).

Today the practical injection path is crafting KV batches by hand (`debug send-kv-batch`). That works, but:

- Transaction records are internal keys; raw Puts are not a supported client write path.
- Encoding real SQL primary-index keys by hand (base64) is awkward for operators targeting an existing table’s keyspace.
- `SendKVBatch` historically rejected transactional batches (`CrossRangeTxnWrapperSender` fatal); that is being fixed so crafted `HeartbeatTxn` / `EndTxn` batches can be sent.

We need a small, purpose-built debug tool that generates synthetic txn records against table/index IDs without requiring encoded keys up front.

**Describe the solution you'd like**

### Command

```text
cockroach debug txn-record-nemesis [flags]
# or stdin when no file is given
```

Connection model matches `debug send-kv-batch`: `--url` / host+certs → Admin RPC (`SendKVBatch`). Not a SQL session.

### What it does (and does not do)

For each txn spec in the input, sequentially send one `BatchRequest` containing a single `HeartbeatTxn`, with:

- `Header.Txn` = fresh PENDING transaction
- `Txn.Key` / request key = resolved anchor key
- `Txn.LockSpans` = resolved lock keys (persisted on first create via `AsRecord()`)

**Does write:** PENDING `TransactionRecord`s that list the claimed lock spans.

**Does not write:** real intents/locks on those keys. Help text must say this explicitly. The stress target is “fat txn records → later GC/abort → IR fanout,” not “ranges full of intents.”

Records persist until abandoned-txn GC (`kv.gc.txn_cleanup_threshold`) pushes/aborts them (or until an explicit abort/commit). No concurrency/batching in v1 — sequential is fine.

### Input: protobuf-backed JSON

Define a small proto (e.g. under `pkg/cli` debug helpers) and accept JSON via `protoutil.JSONPb`, same pattern as `send-kv-batch`.

```protobuf
message TxnRecordNemesisSpec {
repeated TxnRecordNemesisTxn txns = 1;
}

message TxnRecordNemesisTxn {
// Anchor for the txn record (Txn.Key / HeartbeatTxn key).
KeyRef anchor = 1 [(gogoproto.nullable) = false];
// Claimed lock spans persisted on the PENDING record.
repeated KeyRef locks = 2 [(gogoproto.nullable) = false];
}

message KeyRef {
// Exactly one of index or key should be set.
IndexKeyRef index = 1;
bytes key = 2 [(gogoproto.casttype) =
"github.com/cockroachdb/cockroach/pkg/roachpb.Key"];
}

message IndexKeyRef {
uint32 table_id = 1;
uint32 index_id = 2;
// Optional; if unset, tool picks a large random int64.
int64 pk = 3;
}
```

Example:

```json
{
"txns": [
{
"anchor": {"index": {"tableId": 104, "indexId": 1}},
"locks": [
{"index": {"tableId": 104, "indexId": 1}},
{"index": {"tableId": 104, "indexId": 1, "pk": 999001}},
{"key": ""}
]
}
]
}
```

### Key resolution

For `IndexKeyRef`, synthesize a point key in that index’s keyspace:

```text
/Table////0
```

(family 0). This is intentionally **not** general SQL row encoding — it only needs to land in the right index keyspace. Document that assumption. Optional raw `key` is the escape hatch for arbitrary addresses.

Use the codec appropriate to the connected tenant/cluster (same rules as other debug KV tools).

### Prerequisites / related work

- `SendKVBatch` must forward transactional batches to DistSender (bypass `CrossRangeTxnWrapperSender`). Covered by local work / tests around `TestSendKVBatchInjectTxnRecord`.
- Unit/CLI test can mirror the existing HeartbeatTxn flavor in `pkg/cli/debug_send_kv_batch_test.go` (SQL table + encoded keys + QueryTxn to confirm PENDING record + LockSpans).

### v1 non-goals

- Concurrency / parallel injection
- Chunking multiple HeartbeatTxns into one `BatchRequest` (impossible: one `Header.Txn` per batch)
- `EndTxn(commit)` / real intent-resolution stress flavor (follow-up; local keys auto-GC, external keys race with async IR)
- Writing real intents
- Multi-column / non-INT primary-key encoding
- Looking up table IDs by name (operator supplies descriptor/index IDs)

### Follow-ups (optional, out of scope)

- `EndTxn(commit)` mode for IR stress (with guidance on external LockSpans)
- `--count` / per-lock expansion counts for template expansion
- Concurrency

**Describe alternatives you've considered**

1. **Raw `debug send-kv-batch` only** — works once transactional batches are allowed, but encoding keys and assembling HeartbeatTxn JSON by hand does not scale for “tons of locks on some range.”
2. **Store-level / unsafe engine writes of `TransactionRecord`** — bypasses Raft placement rules and is easy to get wrong across replicas; rejected.
3. **`EndTxn(commit)` as the primary injector** — triggers sync/async intent resolution immediately; good for IR stress, bad for leaving stable PENDING records. HeartbeatTxn is the better v1 default; EndTxn can be a later mode.
4. **Name `intentnemesis`** — oversells: we are not writing intents. Prefer `txn-record-nemesis`.

**Additional context**

Relevant internals:

- `HeartbeatTxn` creates a PENDING record from `Header.Txn` on miss (`AsRecord()` includes `LockSpans`); no intent resolution.
- `BeginTransaction` was removed (lazy txn record creation); HeartbeatTxn is the supported “create record” path.
- Local investigation / prototype path: `pkg/cli/debug_send_kv_batch_test.go` (`TestSendKVBatchInjectTxnRecord` / `heartbeat` subtest) constructs SQL primary-index keys and injects via `send-kv-batch`.

Acceptance criteria for v1:

- [ ] `cockroach debug txn-record-nemesis` reads protobuf-JSON spec from file or stdin
- [ ] Sequentially injects PENDING txn records via Admin `SendKVBatch` + `HeartbeatTxn`
- [ ] Supports `IndexKeyRef` (table/index/+optional pk) and raw `key`
- [ ] Help text clearly states: no real intents; records claim lock spans only
- [ ] CLI test covers at least one HeartbeatTxn injection + `QueryTxn` verification

Jira issue: CRDB-65876

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cli/debug_send_kv_batch_test.go, especially TestSendKVBatchInjectTxnRecord and its heartbeat subtest, then trace the existing send-kv-batch Admin RPC path. Implement the protobuf-JSON file or stdin flow and sequential HeartbeatTxn injection for index and raw keys. Done means the CLI test verifies a PENDING record and LockSpans with QueryTxn, and help text states that no real intents are written.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.