cockroachdb / cockroachdb/cockroach

roachtest-operation: rewrite failure injection operations to use roachprod failure injection library

Open
#164,901 3 comments 0 reactions 0 assignees View on GitHub
A-testeng-foundations T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

Rewrite the three failure injection DRT operations (`node-kill`, `network-partition`, `disk-stall`) to use the roachprod failure injection library instead of ad-hoc shell commands. The library provides a structured `FailureMode` interface with proper state management, recovery monitoring, and cleanup safety.

The `cluster.Cluster.GetFailer()` method returns a `*failures.Failer` for any registered failure mode. The disk-stall operation already partially uses this via `roachtestutil.DiskStaller` — use that as a reference pattern.

## Task 1: Rewrite `node-kill` operations

Rewrite `pkg/cmd/roachtest/operations/node_kill.go` to use the `"process-kill"` failure mode from `pkg/roachprod/failureinjection/failures/process_kill.go`. Re-register the four existing operation variants and uncomment `registerNodeKill(r)` in `register.go`.

## Task 2: Rewrite `network-partition` operations

Rewrite `pkg/cmd/roachtest/operations/network_partition.go` to use the `"iptables-network-partition"` failure mode from `pkg/roachprod/failureinjection/failures/network_partition.go`. Keep the same two operation names.

## Task 3: Align `disk-stall` cleanup with library

`pkg/cmd/roachtest/operations/disk_stall.go` already uses `roachtestutil.DiskStaller` which wraps the library, but cleanup has manual node-restart logic that duplicates what the library provides. Remove the duplication and let the library handle recovery.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.