(m)TLS replication is broken in 2.9.0
- Dominant language
- C++
- Stars
- 4.4k
- Forks
- 658
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 10
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
### Version
2.9.0
### Minimal reproduce step
When upgrading Kvrocks from 2.8.0 to 2.9.0, we started to get SSL/TLS errors when trying to connect a slave to the master. No problem on replication without TLS.
Both master and slave are on 2.9.0. Rolling back the master to 2.8.0 and keeping the replica on 2.9.0 is working so it is definitely on the "server/master" part.
When both were on 2.9.0, using `redis-cli` on the slave instance to connect to the master was working with the certificates so they are fine:
No errors (on replica instance):
```
redis-cli -h kvrocks-master \
-p 6379 \
--tls \
--cacert /ca/kvrocks/ca.crt \
--cert /tls/kvrocks/tls.crt \
--key /tls/kvrocks/tls.key
```
Errors (replica instance, see below):
```
kvrocks -c kvrocks.conf \
--dir /var/lib/kvrocks \
--pidfile /var/run/kvrocks/kvrocks.pid \
--masterauth "xxx" \
--slaveof "kvrocks-master 6379" \
--tls-ca-cert-file /ca/kvrocks/ca.crt \
--tls-key-file /tls/kvrocks/tls.key \
--tls-cert-file /tls/kvrocks/tls.crt \
--tls-replication yes \
--bind 0.0.0.0
```
### What did you expect to see?
A working (m)TLS replication that either does psync or full synchronization
### What did you see instead?
Server (MASTER) :
> kvrocks I20240813 08:37:26.913834 121 cmd_replication.cc:60] Slave 100.65.46.20:45098, listening port: 6379, announce ip: 100.65.46.20 asks for synchronization with next sequence: 1 replication id: not supported, and local sequence: 344837857
kvrocks E20240813 08:37:26.918999 121 redis_connection.cc:109] [connection] Going to remove the client: 100.65.46.20:45098, while encounter error: Success, SSL Error: error:0A000126:SSL routines::unexpected eof while reading
kvrocks I20240813 08:37:26.986922 193 cmd_replication.cc:242] [replication] Succeed sending full data file info to 100.65.46.20
kvrocks W20240813 08:37:27.038514 194 cmd_replication.cc:299] [replication] Fail to send file CURRENT to 100.65.46.20, error: Success
kvrocks I20240813 08:37:37.086854 195 cmd_replication.cc:242] [replication] Succeed sending full data file info to 100.65.46.20
kvrocks W20240813 08:37:37.127951 196 cmd_replication.cc:299] [replication] Fail to send file CURRENT to 100.65.46.20, error: Success
Client (REPLICA) :
> W20240813 08:00:12.653694 50 replication.cc:935] [fetch] Fail to fetch file 005813.sst, err: fetch file err: read sst file: failed to read from SSL connection: error:00000000:lib(0)::reason(0)
W20240813 08:00:12.655525 49 replication.cc:935] [fetch] Fail to fetch file 009665.sst, err: fetch file err: read sst file: failed to read from SSL connection: error:00000000:lib(0)::reason(0)
W20240813 08:00:12.660212 51 replication.cc:935] [fetch] Fail to fetch file 008792.sst, err: fetch file err: read sst file: failed to read from SSL connection: error:00000000:lib(0)::reason(0)
W20240813 08:00:12.661721 52 replication.cc:935] [fetch] Fail to fetch file 005736.sst, err: fetch file err: read sst file: failed to read from SSL connection: error:00000000:lib(0)::reason(0)
### Anything Else?
Is it safe to downgrade to 2.8.0 on instances where I need (m)tls replication? Could it be due to the switch to Debian?
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the 2.8.0-to-2.9.0 upgrade with the provided TLS replication configuration and compare the master and replica errors. Trace the replication paths in cmd_replication.cc and replication.cc, along with SSL connection handling in redis_connection.cc. Done means both PSYNC and full synchronization work over TLS without the reported connection errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100