apache / apache/incubator-pegasus

Bug(go client):The cluster added a new meta node, but the meta server configuration of the go client was not updated. As a result, the client cannot find the new meta address and can only access the meta listed in the meta list

Open
#1,880 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

Assuming the Pegasus client is configured with a meta server list of "127.0.0.1:34602" and "127.0.0.1:34603," but the actual primary meta server for the Pegasus server is "127.0.0.1:34601," the Pegasus client will not be able to connect to the Pegasus server until a timeout occurs.

The reason is that when the go client searches for the primary, it iterates through the meta server list, sending an RPC RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX to each meta server and making a determination based on the response.

Unlike the Java client, the go client cannot directly use indirection to add meta servers not specified in the configuration to the client.
Below is the logic code for this part of the go client.

// go-client/session/meta_call.go  
func (c *metaCall) issueBackupMetas(ctx context.Context) {
    for i := range c.metas {
       if i == c.lead {
          continue
       }
       // concurrently issue RPC to the rest of meta servers.
       go func(idx int) {
          c.issueSingleMeta(ctx, idx)
       }(i)
    }
}

// issueSingleMeta returns false if we should try another meta
func (c *metaCall) issueSingleMeta(ctx context.Context, i int) bool {
    meta := c.metas[i]
    resp, err := c.callFunc(ctx, meta)
    if err != nil || resp.GetErr().Errno == base.ERR_FORWARD_TO_OTHERS.String() {
       return false
    }
    // the RPC succeeds, this meta becomes the new leader now.
    atomic.StoreUint32(&c.newLead, uint32(i))
    select {
    case <-ctx.Done():
    case c.respCh <- resp:
       // notify the caller
    }
    return true
}

Here is the relevant part of the Java client code for this:

// com/xiaomi/infra/pegasus/rpc/async/MetaSession.java  onFinishQueryMeta()

synchronized (this) {
  if (needSwitchLeader) {
    if (forwardAddress != null && !forwardAddress.isInvalid()) {
      boolean found = false;
      for (int i = 0; i < metaList.size(); i++) {
        if (metaList.get(i).getAddress().equals(forwardAddress)) {
          curLeader = i;
          found = true;
          break;
        }
      }
      if (!found) {
        logger.info("add forward address {} as meta server", forwardAddress);
        metaList.add(clusterManager.getReplicaSession(forwardAddress));
        curLeader = metaList.size() - 1;
      }
    } else if (metaList.get(curLeader) == round.lastSession) {
      curLeader = (curLeader + 1) % metaList.size();
      if (curLeader == 0 && hostPort != null && round.maxResolveCount != 0) {
        resolveHost(hostPort);
        round.maxResolveCount--;
        round.maxExecuteCount = metaList.size();
      }
    }
  }
  round.lastSession = metaList.get(curLeader);
}
In summary

The primary impact of this issue is that, in the online cluster, a new meta server was added, and at some point thereafter, this meta server became the primary. Users, without changing their configurations, are unable to connect to the server.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with go-client/session/meta_call.go, especially issueBackupMetas and issueSingleMeta, then compare the forwarding-address handling in com/xiaomi/infra/pegasus/rpc/async/MetaSession.java. Trace how a successful response identifies a primary outside the configured list. Done means the Go client can discover and use a newly added primary meta server without changing its configured meta list.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.