redpanda-data / redpanda-data/redpanda
cloud_storage/scrubbing: heavy weight scrubbing
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 12.6k
- Forks
- 792
- PR merge metrics
- No merged PRs in 30d
Description
Currently, the scrubber discovers anomalies solely by looking at the metadata and validating that referenced objects exist in cloud storage. This is a great start, but it could be taken further by downloading referenced objects (e.g. segments) and validating that the metadata checks out with them. Note that this can be very heavyweight in terms of resources and costs. Hence, choosing whether to perform a "deep scrub" (i.e. download and check batches/records) could be delegated to a probabilistic model that's aware of S3 costs.
JIRA Link: CORE-1586
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The title identifies the cloud_storage/scrubbing subsystem, but no files, tests, or entry points are named. Start by locating the scrubber implementation and its existing metadata and object-existence checks. Done would require a defined deep-scrub path that downloads referenced objects, validates their metadata, and determines when to run it using an S3-cost-aware probabilistic model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100