apache / apache/servicecomb-java-chassis

在SafeMode下如果pullInstances成功后无法在退出SafeMode后老化旧的实例

Open
#2,820 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.9k
Forks
814
Avg merge
8d 23h
Merged PRs (30d)
1

Description

**Describe the bug**
背景:servicecomb-java-chassis 2.1.5在使用过程中,我们的rpc调用服务端会校验客户端在请求头中携带的目标实例ID,发现某些rpc请求的实例ID对应已经老化的实例ID。
现象:服务缓存中存在已经老化的旧实例。
原因分析:RefreshableMicroserviceCache缓存会在pullInstance后更新记录的revisionId,如果在safemode期间pullInstance成功了会导致此处rev被更新的和sc中的一致,在退出安全模式后携带此rev再次pullInstance时如果此时实例在sc中未有变更会直接响应304,不会触发缓存刷新,导致在安全模式中不老化的实例无法正常下线。
解决建议:在安全模式下不更新缓存中记录的revisionId。
**To Reproduce**
复现方法:参考以下测试用例。
_RefreshableMicroserviceCacheTest.java_
```
@Test
public void refresh_exit_safe_mode_down_old_instance() {
ArrayList instances = new ArrayList<>();
findServiceInstancesOprHolder.value = params -> {
Assert.assertEquals("consumerId", params[0]);
Assert.assertEquals("app", params[1]);
Assert.assertEquals("svc", params[2]);
Assert.assertEquals("0.0.0.0+", params[3]);
MicroserviceInstances microserviceInstances = new MicroserviceInstances();
microserviceInstances.setNeedRefresh(!String.valueOf(instances.hashCode()).equals(params[4]));
microserviceInstances.setRevision(String.valueOf(instances.hashCode()));
microserviceInstances.setMicroserviceNotExist(false);

FindInstancesResponse instancesResponse = new FindInstancesResponse();
instancesResponse.setInstances(instances);

microserviceInstances.setInstancesResponse(instancesResponse);
return microserviceInstances;
};

// at the beginning, no instances in cache
List cachedInstances = microserviceCache.getInstances();
Assert.assertEquals(0, cachedInstances.size());
Assert.assertNull(microserviceCache.getRevisionId());

// find 1 instance from sc
MicroserviceInstance microserviceInstance = new MicroserviceInstance();
instances.add(microserviceInstance);
microserviceInstance.setInstanceId("instanceId00");

microserviceCache.refresh();
Assert.assertEquals(MicroserviceCacheStatus.REFRESHED, microserviceCache.getStatus());

cachedInstances = microserviceCache.getInstances();
Assert.assertEquals(1, cachedInstances.size());
MicroserviceInstance instance = cachedInstances.iterator().next();
Assert.assertEquals("instanceId00", instance.getInstanceId());
Assert.assertEquals("-738005329", microserviceCache.getRevisionId());

// 2nd time, clear pull instances
MicroserviceInstance microserviceInstance1 = new MicroserviceInstance();
instances.clear();
instances.add(microserviceInstance1);
microserviceInstance1.setInstanceId("instanceId01");

microserviceCache.refresh();
Assert.assertEquals(MicroserviceCacheStatus.REFRESHED, microserviceCache.getStatus());
cachedInstances = microserviceCache.getInstances();
Assert.assertEquals(1, cachedInstances.size());
Assert.assertEquals("instanceId01", cachedInstances.get(0).getInstanceId());

// 3nd time, 进入安全模式,第三次拉取不下线实例
microserviceCache.onSafeModeChanged(new SafeModeChangeEvent(true));

MicroserviceInstance microserviceInstance2 = new MicroserviceInstance();
instances.clear();
instances.add(microserviceInstance2);
microserviceInstance2.setInstanceId("instanceId02");

microserviceCache.refresh();
Assert.assertEquals(MicroserviceCacheStatus.REFRESHED, microserviceCache.getStatus());

cachedInstances = microserviceCache.getInstances();
Assert.assertEquals(2, cachedInstances.size());
Assert.assertEquals("instanceId02", cachedInstances.get(0).getInstanceId());
Assert.assertEquals("instanceId01", cachedInstances.get(1).getInstanceId());

// 4nd time, 退出安全模式,第四次拉取需要下线实例
microserviceCache.onSafeModeChanged(new SafeModeChangeEvent(false));

microserviceCache.refresh();
Assert.assertEquals(MicroserviceCacheStatus.REFRESHED, microserviceCache.getStatus());

cachedInstances = microserviceCache.getInstances();
Assert.assertEquals(1, cachedInstances.size());
Assert.assertEquals("instanceId02", cachedInstances.get(0).getInstanceId());
}
```

**Expected behavior**
在退出安全模式后可以及时下线老实例。

**Platform And Runtime (please complete the following information):**
环境无差别,稳定复现

**Additional context**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with RefreshableMicroserviceCache and the named RefreshableMicroserviceCacheTest.java test, especially refresh() and safe-mode state changes. Reproduce the sequence in refresh_exit_safe_mode_down_old_instance: after leaving safe mode, the previously cached instance should be removed and only the current instance should remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.