eBay / eBay/HomeStore

handle create repl_dev failure

Open
#807 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
29
Forks
30
Avg merge
2d 4h
Merged PRs (30d)
6

Description

Creating a replication device (repl dev) can fail and leave garbage repl devs on certain members. For example:

  • Leader adds member F2, but the operation times out.
  • Leader assumes F2 is not in the group, but F2 successfully joins.
  • When the group is destroyed, F2 is not included, leaving an orphaned group on F2.
Related Logs

Leader:

  • Sent join request to F2:
    [09/20/25 09:01:12.178923] [I] [76] [handle_join_leave.cxx:149:invite_srv_to_join_cluster] sent join request to peer 984278082, 98f2b032-095d-4d78-a069-d1e21f95603d [group=e3be8382-63f2-4edd-85ed-8851e8e65641]
    
  • Timeout occurred:
    [09/20/25 09:01:14.179209] [I] [61] [raft_server.cxx:1639:handle_ext_resp_err] receive an rpc error response from peer server, Deadline Exceeded 12 [group=e3be8382-63f2-4edd-85ed-8851e8e65641]
    

Follower (F2):

  • Received the join request after the leader timed out:
    [09/20/25 09:01:14.390810] [I] [65] [handle_join_leave.cxx:188:handle_join_cluster_req] got join cluster req from leader 2114978300 [group=e3be8382-63f2-4edd-85ed-8851e8e65641]
    

For more context, refer to the related discussion: GitHub PR #136 - Discussion.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the discussion in PR #136, then inspect handle_join_leave.cxx around invite_srv_to_join_cluster and handle_join_cluster_req, alongside the timeout path in raft_server.cxx. Trace the timeout and late-join sequence described in the logs. Done means a failed repl dev creation does not leave an orphaned group or device on a member after cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.