eBay / eBay/HomeObject

Clean up stale resources when BR destroy PG

Open
#394 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
11
Forks
17
Avg merge
4d 5h
Merged PRs (30d)
5

Description

SH tests found 2 issues: When a member is resynced by BR, all PG resources (blob allocation/logs/status/...) are expected to be reset/new. However, pg_destroy misses some resources, leaving stale state, which can cause corrupted blobs or system crashes. Here we need:

  1. Clean up all currently identified stale resources;
  2. Discuss how to ensure future resources are reliably cleaned up when BR destroys a PG, since there might be more resources added in the future.

Currently there are two resources that need cleanup during BR PG destroy, related to 2 issues:

  1. stale rreqs (SH issue#91)
    The chunk block index is reset by BR, but the associated rreqs in m_repl_key_req_map are not cleared. After BR completes, when the log is appended from Raft, the stale rreq in memory is reused, leading to an incorrect block.
  2. no_space_left_error_info (SH issue#95)
    EGC handle_no_space_left waits for commits up to LSN1, then BR occurs and advances commit_lsn to LSN2 (LSN2 > LSN1). Since the stale no_space_left_error_info is not reset and its lsn < commit_lsn, this hits the assert in notify_committed_lsn called by flush_durable_commit_lsn

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the BR PG destroy path, then inspect m_repl_key_req_map and no_space_left_error_info. Read replication_state_machine.cpp around notify_committed_lsn and the handle_no_space_left flow, including flush_durable_commit_lsn. Done means the identified stale resources are reset during destroy and the SH scenarios no longer produce corrupted state or assertion failures, with a reliable approach for future resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.