operation/failureinjection: OOD (out of disk)

Similar to OOM [1], we need to regularly test for out of disk failures. In some cases, pebble will panic the node, thus failing fast. In other cases, the node may remain partially available for an arbitrary duration.

A recent outage in `drt-chaos` [2] resulted from an online-restore job filling up disks on two different nodes. While no (replicated) data was corrupted, the cluster exhibited (partial) unavailability. Further, while the nodes continued to run, the logs were spammed with the error of the form,

```
E260120 18:17:24.433674 156859 3@pebble/event.go:1224 ⋮ [n1,s2,pebble] 11098 background error: pebble: backing file 23383613 error: ‹storage: object doesn't exist›
```

Upon restarting a node, `replicaGC` would panic,

```
panic: error during replicaGC: pebble: backing file 23376765 error: storage: object doesn't exist

goroutine 168448 gp=0xc0d92ad4a0 m=26 mp=0xc0462d5008 [running, "n":"1", "replicaGC":""]:
panic({0x727c940?, 0xc028ea2b70?})
GOROOT/src/runtime/panic.go:802 +0x168 fp=0xc0538c96b0 sp=0xc0538c9600 pc=0x4f0d88
github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic({0x88ef4c8, 0xc02b5a6fc0}, 0xc00374e000, {0x791f94a?, 0x15?}, {0xc0538c9830?, 0x0?, 0x0?})
pkg/util/log/logcrash/crash_reporting.go:434 +0x1bc fp=0xc0538c9730 sp=0xc0538c96b0 pc=0xf9bdbc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaGCQueue).process(0xc012abb3c8, {0x88ef4c8, 0xc02b5a6fc0}, 0xc01c384388, {0xd?, 0x0?}, 0x0?)
pkg/kv/kvserver/replica_gc_queue.go:316 +0x349 fp=0xc0538c9890 sp=0xc0538c9730 pc=0x47d3d09
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1({0x88ef4c8, 0xc02b5a6fc0})
pkg/kv/kvserver/queue.go:1255 +0x8bc fp=0xc0538c9cb0 sp=0xc0538c9890 pc=0x4771b3c
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x88ef768, 0xc052f750e0}, {0xc025112f00, 0x2d}, 0xdf8475800, 0xc0538c9e40)
pkg/util/timeutil/timeout.go:28 +0xd7 fp=0xc0538c9d50 sp=0xc0538c9cb0 pc=0x9d69d7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica(0xc00da81200, {0x88ef768, 0xc052f750e0}, {0x8951200, 0xc01c384388}, 0x3ff0000000000000)
pkg/kv/kvserver/queue.go:1248 +0x47c fp=0xc0538c9ea0 sp=0xc0538c9d50 pc=0x47711bc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processOneAsyncAndReleaseSem.func1({0x88ef768, 0xc052f750e0})
pkg/kv/kvserver/queue.go:1196 +0xd4 fp=0xc0538c9f58 sp=0xc0538c9ea0 pc=0x4770b14
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x88ef768, 0xc052f750e0})
pkg/util/stop/stopper.go:440 +0x8a fp=0xc0538c9fc0 sp=0xc0538c9f58 pc=0x14f4a2a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
pkg/util/stop/stopper.go:441 +0x28 fp=0xc0538c9fe0 sp=0xc0538c9fc0 pc=0x14f4968
runtime.goexit({})
src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0538c9fe8 sp=0xc0538c9fe0 pc=0x4fba61
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 3354
pkg/util/stop/stopper.go:438 +0x125
```

The above suggests `ENOSPC` left pebble in an inconsistent state. There might be multiple root causes, hence the impetus for this story.

[1] https://github.com/cockroachdb/cockroach/issues/149161
[2] https://github.com/cockroachdb/cockroach/issues/161834

Jira issue: CRDB-59097