apache / apache/incubator-pegasus
Bug(go client):The cluster added a new meta node, but the meta server configuration of the go client was not updated. As a result, the client cannot find the new meta address and can only access the meta listed in the meta list
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
Assuming the Pegasus client is configured with a meta server list of "127.0.0.1:34602" and "127.0.0.1:34603," but the actual primary meta server for the Pegasus server is "127.0.0.1:34601," the Pegasus client will not be able to connect to the Pegasus server until a timeout occurs.
The reason is that when the go client searches for the primary, it iterates through the meta server list, sending an RPC RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX to each meta server and making a determination based on the response.
Unlike the Java client, the go client cannot directly use indirection to add meta servers not specified in the configuration to the client.
Below is the logic code for this part of the go client.
// go-client/session/meta_call.go
func (c *metaCall) issueBackupMetas(ctx context.Context) {
for i := range c.metas {
if i == c.lead {
continue
}
// concurrently issue RPC to the rest of meta servers.
go func(idx int) {
c.issueSingleMeta(ctx, idx)
}(i)
}
}
// issueSingleMeta returns false if we should try another meta
func (c *metaCall) issueSingleMeta(ctx context.Context, i int) bool {
meta := c.metas[i]
resp, err := c.callFunc(ctx, meta)
if err != nil || resp.GetErr().Errno == base.ERR_FORWARD_TO_OTHERS.String() {
return false
}
// the RPC succeeds, this meta becomes the new leader now.
atomic.StoreUint32(&c.newLead, uint32(i))
select {
case <-ctx.Done():
case c.respCh <- resp:
// notify the caller
}
return true
}
Here is the relevant part of the Java client code for this:
// com/xiaomi/infra/pegasus/rpc/async/MetaSession.java onFinishQueryMeta()
synchronized (this) {
if (needSwitchLeader) {
if (forwardAddress != null && !forwardAddress.isInvalid()) {
boolean found = false;
for (int i = 0; i < metaList.size(); i++) {
if (metaList.get(i).getAddress().equals(forwardAddress)) {
curLeader = i;
found = true;
break;
}
}
if (!found) {
logger.info("add forward address {} as meta server", forwardAddress);
metaList.add(clusterManager.getReplicaSession(forwardAddress));
curLeader = metaList.size() - 1;
}
} else if (metaList.get(curLeader) == round.lastSession) {
curLeader = (curLeader + 1) % metaList.size();
if (curLeader == 0 && hostPort != null && round.maxResolveCount != 0) {
resolveHost(hostPort);
round.maxResolveCount--;
round.maxExecuteCount = metaList.size();
}
}
}
round.lastSession = metaList.get(curLeader);
}
In summary
The primary impact of this issue is that, in the online cluster, a new meta server was added, and at some point thereafter, this meta server became the primary. Users, without changing their configurations, are unable to connect to the server.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with go-client/session/meta_call.go, especially issueBackupMetas and issueSingleMeta, then compare the forwarding-address handling in com/xiaomi/infra/pegasus/rpc/async/MetaSession.java. Trace how a successful response identifies a primary outside the configured list. Done means the Go client can discover and use a newly added primary meta server without changing its configured meta list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100