cockroachdb / cockroachdb/cockroach
storage,kv: use delayed deletion of old state in some roachtests
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Inspired by https://github.com/cockroachdb/cockroach/issues/143058.
When roachtests fail with a data consistency error, investigations usually go nowhere since the issue won't or is prohibitively expensive to reproduce, and since we no longer have the dataset.
In past reproduction cycles, we've used `pebble.ArchiveCleaner` which instructs pebble to move files to an archive folder (instead of deleting).
We could run the set of non-benchmark roachtests with a more powerful version of this cleaner that also garbage-collect files once disk usage reaches (say) 80%. In fact, we could hack this toegether without any changes to archive cleaner, too, by spawning (via systemd) a cron job that checks disk usage every minute and deletes the oldest files until under threshold.
Jira issue: CRDB-48767
Contributor guide
Assessment
This issue has not been assessed yet.