redpanda-data / redpanda-data/redpanda

Suppress `404 Not Found` errors on read replicas

Open
#8,965 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

area/cloud-storage kind/enhance
Dominant language
C++
Stars
12.6k
Forks
792
PR merge metrics
No merged PRs in 30d

Description

Version & Environment

Redpanda version: (use rpk version): 22.2.7

What went wrong?

After creating a read replica topic, broker logs showed that remote manifest files were being looked for that didn't exist in the bucket, and this resulted in 404 Not Found.

WARN  2023-02-16 20:35:22,052 [shard 11] s3 - client.cc:513 - S3 replied with error: HTTP/1.1 404 Not Found
X-GUploader-UploadID: ADPycduQPcpaVk8asFCR9DT516kd4huFe4tsPAg3UupRUlYBF69peeCOKQ0E6MEl1z4WsGZTm9y2ss6rRwJbwvO5TI0GVvgKy7y0
Content-Type: application/xml; charset=UTF-8
Content-Length: 238
Date: Thu, 16 Feb 2023 20:35:22 GMT
Expires: Thu, 16 Feb 2023 20:35:22 GMT
Cache-Control: private, max-age=0
Server: UploadServer


INFO  2023-02-16 20:35:22,052 [shard 11] cloud_storage - [fiber74~6~0|1|9931ms] - remote.cc:74 - NoSuchKey response received {"90000000/meta/kafka/pippio-test/12_70/manifest.json"}
ERROR 2023-02-16 20:35:22,052 [shard 11] archival - [fiber74 kafka/pippio-test/12] - ntp_archiver_service.cc:270 - Failed to download partition manifest in read-replica mode
ERROR 2023-02-16 20:35:22,052 [shard 11] archival - [fiber74 kafka/pippio-test/12] - ntp_archiver_service.cc:253 - Failed to download manifest {"90000000/meta/kafka/pippio-test/12_70/manifest.json"}

The reason for these messages is because the filenames are deterministic, so the remote cluster looks for manifest files that could exist (not the files that the source cluster have been confirmed to have generated).

What should have happened instead?

The source cluster should not throw errors in the log related to not finding manifest files that shouldn't exist, as users understandably assume this means there is an issue they need to track down. Instead, there should be a way for the remote cluster to only look for files that should exist given the state of the source cluster and bucket.

How to reproduce the issue?
  1. create a read replica topic
  2. view broker logs
Additional information

Attached relevant broker log:
rrr-error.log

Internal slack discussion around this issue: https://redpandadata.slack.com/archives/C02BDN76HUK/p1676648517086209

JIRA Link: CORE-1168

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the read-replica flow by creating a read replica topic and reviewing broker logs. Start at the reported locations in ntp_archiver_service.cc, remote.cc, and client.cc, then trace the manifest lookup behavior. Done means expected missing deterministic manifest files no longer produce misleading errors while genuine failures remain visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.