CodisLabs / CodisLabs/codis

Codis-HA的使用问题

Open
#1,462 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.2k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

在使用codis-ha时出现如下问题,不知是否正常。

**场景如下**:
启用codis-ha后,当一个组内的master节点down掉后,codis-ha会选举group内的一个slave节点作为master节点。**但随即会剔除group内除新提升为master的节点以外,其他所有节点都会被强制剔除且服务器上确认codis-server进程退出。** 当手动启动被剔除的节点后,再次添加到该group依然无法添加(添加后会被自动剔除且进程自动退出)。

**此外,想要了解一些问题**:
1. 想要了解codis-ha的实现机制,是基于redis-sentinel实现的吗?
2. codis-ha在使用时,需要另外以sentinel的模式运行redis吗?
3. codis-ha的主从切换是怎样的?
4. 后续的数据同步的流程(当原master重启上线再加入到group中时)是怎样的呢?

**望解答,谢谢。**

----

#### 相关信息如下
**group组实例**
![snipaste_20180319_163447.jpg](https://i.loli.net/2018/03/19/5aaf76c082303.jpg)

6181为master,6282为slave。

**Codis-server日志信息**
6381
> 8541:signal-handler (1521448920) Received SIGTERM scheduling shutdown...
8541:M 19 Mar 16:42:00.622 # User requested shutdown...
8541:M 19 Mar 16:42:00.622 * Removing the pid file.
8541:M 19 Mar 16:42:00.623 # Redis is now ready to exit, bye bye...

6282
>1002:S 19 Mar 16:42:00.625 # Connection with master lost.
1002:S 19 Mar 16:42:00.626 * Caching the disconnected master state.
1002:S 19 Mar 16:42:00.630 * Connecting to MASTER 10.1.60.72:6381
1002:S 19 Mar 16:42:00.631 * MASTER <-> SLAVE sync started
1002:S 19 Mar 16:42:00.631 # Error condition on socket for SYNC: Connection refused
1002:S 19 Mar 16:42:01.632 * Connecting to MASTER 10.1.60.72:6381
1002:S 19 Mar 16:42:01.632 * MASTER <-> SLAVE sync started
1002:S 19 Mar 16:42:01.632 # Error condition on socket for SYNC: Connection refused
1002:M 19 Mar 16:42:01.729 * Discarding previously cached master state.
1002:M 19 Mar 16:42:01.730 * MASTER MODE enabled (user request from 'id=307 addr=10.1.60.71:49794 fd=5 name= age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=0 qbuf-free=32768 obl=50 oll=0 omem=0 events=r cmd=exec')
1002:M 19 Mar 16:42:01.730 # CONFIG REWRITE executed with success.

**Codis-HA日志如下(日志级别Debug)**

> 2018/03/19 16:42:01 api.go:139: [DEBUG] call rpc [GET] http://10.1.60.71:18080/api/topom/stats/95b62887719520f17e312eaa76d28f2b in 12.590253ms
2018/03/19 16:42:01 main.go:206: [INFO] [ ] proxy-1 [T] 73c66af68de27c782306597171dd8a35 [A] 10.1.60.71:11080 [P] 10.1.60.71:19000
2018/03/19 16:42:01 main.go:206: [INFO] [ ] proxy-2 [T] 0fd8a1488d9b366dac8e09ac5235e4c2 [A] 10.1.60.72:11080 [P] 10.1.60.72:19000
2018/03/19 16:42:01 main.go:206: [INFO] [ ] proxy-3 [T] e3619878e11ed86d6c0d378f9b0a918e [A] 10.1.60.73:11080 [P] 10.1.60.73:19000
2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-1 [0] 10.1.60.71:6379
2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-1 [1] 10.1.60.71:6380
2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-2 [0] 10.1.60.72:6379
2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-2 [1] 10.1.60.72:6380
2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-3 [0] 10.1.60.73:6379
2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-3 [1] 10.1.60.73:6380
2018/03/19 16:42:01 main.go:236: [WARN] [E] group-4 [0] 10.1.60.72:6381
2018/03/19 16:42:01 main.go:242: [WARN] [X] group-4 [1] 10.1.60.72:6382
2018/03/19 16:42:01 main.go:274: [WARN] codis-server (master) 10.1.60.72:6381 state error
2018/03/19 16:42:01 main.go:360: [WARN] try to promote group-[4] with slave 10.1.60.72:6382
2018/03/19 16:42:01 api.go:139: [DEBUG] call rpc [PUT] http://10.1.60.71:18080/api/topom/group/promote/95b62887719520f17e312eaa76d28f2b/4/10.1.60.72:6382 in 31.323133ms
2018/03/19 16:42:01 main.go:364: [WARN] done.

> 2018/03/19 16:42:04 api.go:139: [DEBUG] call rpc [GET] http://10.1.60.71:18080/api/topom/stats/95b62887719520f17e312eaa76d28f2b in 13.616047ms
2018/03/19 16:42:04 main.go:206: [INFO] [ ] proxy-1 [T] 73c66af68de27c782306597171dd8a35 [A] 10.1.60.71:11080 [P] 10.1.60.71:19000
2018/03/19 16:42:04 main.go:206: [INFO] [ ] proxy-2 [T] 0fd8a1488d9b366dac8e09ac5235e4c2 [A] 10.1.60.72:11080 [P] 10.1.60.72:19000
2018/03/19 16:42:04 main.go:206: [INFO] [ ] proxy-3 [T] e3619878e11ed86d6c0d378f9b0a918e [A] 10.1.60.73:11080 [P] 10.1.60.73:19000
2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-1 [0] 10.1.60.71:6379
2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-1 [1] 10.1.60.71:6380
2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-2 [0] 10.1.60.72:6379
2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-2 [1] 10.1.60.72:6380
2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-3 [0] 10.1.60.73:6379
2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-3 [1] 10.1.60.73:6380
2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-4 [0] 10.1.60.72:6382
2018/03/19 16:42:04 main.go:236: [WARN] [E] group-4 [1] 10.1.60.72:6381
2018/03/19 16:42:04 main.go:286: [WARN] try to group-del-server to dashboard 10.1.60.72:6381
2018/03/19 16:42:04 api.go:139: [DEBUG] call rpc [PUT] http://10.1.60.71:18080/api/topom/group/del/95b62887719520f17e312eaa76d28f2b/4/10.1.60.72:6381 in 2.467563ms
2018/03/19 16:42:04 main.go:291: [DEBUG] call rpc group-del-server OK
2018/03/19 16:42:04 main.go:294: [WARN] try to shutdown codis-server(slave) 10.1.60.72:6381
2018/03/19 16:42:04 main.go:297: [WARN] connect to codis-server(slave) 10.1.60.72:6381 failed
[error]: dial tcp 10.1.60.72:6381: getsockopt: connection refused
2 /usr/local/codis/src/github.com/CodisLabs/codis/pkg/utils/redis/client.go:42
github.com/CodisLabs/codis/pkg/utils/redis.NewClient
1 /usr/local/codis/src/github.com/CodisLabs/codis/cmd/ha/main.go:295
main.(*HealthyChecker).Maintains
0 /usr/local/codis/src/github.com/CodisLabs/codis/cmd/ha/main.go:95
main.main
... ...

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.