cockroachdb / cockroachdb/cockroach

storage,kv: use delayed deletion of old state in some roachtests

Open
#143,346 0 comments 0 reactions 0 assignees View on GitHub
A-testing C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Inspired by https://github.com/cockroachdb/cockroach/issues/143058.

When roachtests fail with a data consistency error, investigations usually go nowhere since the issue won't or is prohibitively expensive to reproduce, and since we no longer have the dataset.

In past reproduction cycles, we've used `pebble.ArchiveCleaner` which instructs pebble to move files to an archive folder (instead of deleting).

We could run the set of non-benchmark roachtests with a more powerful version of this cleaner that also garbage-collect files once disk usage reaches (say) 80%. In fact, we could hack this toegether without any changes to archive cleaner, too, by spawning (via systemd) a cron job that checks disk usage every minute and deletes the oldest files until under threshold.

Jira issue: CRDB-48767

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.