redpanda-data / redpanda-data/redpanda
Suppress `404 Not Found` errors on read replicas
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 12.6k
- Forks
- 792
- PR merge metrics
- No merged PRs in 30d
Description
Version & Environment
Redpanda version: (use rpk version): 22.2.7
What went wrong?
After creating a read replica topic, broker logs showed that remote manifest files were being looked for that didn't exist in the bucket, and this resulted in 404 Not Found.
WARN 2023-02-16 20:35:22,052 [shard 11] s3 - client.cc:513 - S3 replied with error: HTTP/1.1 404 Not Found
X-GUploader-UploadID: ADPycduQPcpaVk8asFCR9DT516kd4huFe4tsPAg3UupRUlYBF69peeCOKQ0E6MEl1z4WsGZTm9y2ss6rRwJbwvO5TI0GVvgKy7y0
Content-Type: application/xml; charset=UTF-8
Content-Length: 238
Date: Thu, 16 Feb 2023 20:35:22 GMT
Expires: Thu, 16 Feb 2023 20:35:22 GMT
Cache-Control: private, max-age=0
Server: UploadServer
INFO 2023-02-16 20:35:22,052 [shard 11] cloud_storage - [fiber74~6~0|1|9931ms] - remote.cc:74 - NoSuchKey response received {"90000000/meta/kafka/pippio-test/12_70/manifest.json"}
ERROR 2023-02-16 20:35:22,052 [shard 11] archival - [fiber74 kafka/pippio-test/12] - ntp_archiver_service.cc:270 - Failed to download partition manifest in read-replica mode
ERROR 2023-02-16 20:35:22,052 [shard 11] archival - [fiber74 kafka/pippio-test/12] - ntp_archiver_service.cc:253 - Failed to download manifest {"90000000/meta/kafka/pippio-test/12_70/manifest.json"}
The reason for these messages is because the filenames are deterministic, so the remote cluster looks for manifest files that could exist (not the files that the source cluster have been confirmed to have generated).
What should have happened instead?
The source cluster should not throw errors in the log related to not finding manifest files that shouldn't exist, as users understandably assume this means there is an issue they need to track down. Instead, there should be a way for the remote cluster to only look for files that should exist given the state of the source cluster and bucket.
How to reproduce the issue?
- create a read replica topic
- view broker logs
Additional information
Attached relevant broker log:
rrr-error.log
Internal slack discussion around this issue: https://redpandadata.slack.com/archives/C02BDN76HUK/p1676648517086209
JIRA Link: CORE-1168
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the read-replica flow by creating a read replica topic and reviewing broker logs. Start at the reported locations in ntp_archiver_service.cc, remote.cc, and client.cc, then trace the manifest lookup behavior. Done means expected missing deterministic manifest files no longer produce misleading errors while genuine failures remain visible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100