dragonflydb / dragonflydb/dragonfly
test_disconnect_replica failed: master crashes with SIGSEGV when replicas are force-killed during stable sync
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
### Environment
* **Test:** `test_disconnect_replica[df_factory0-df_seeder_factory0-4-t_crash_fs1-t_crash_ss1-t_disonnect1-2000]`
* **File:** `dragonfly/replication_test.py`
* **Environment:** CI (epoll regression tests, `--force_epoll=true`, debug build)
* **Commit:** `01e8e60e0cad7da29eba6ff3517a177bf19ab764`
### Executive Summary
The **master** Dragonfly process (pid `23093`, port `40015`) crashed with **SIGSEGV (exit code -11)** approximately 93 seconds into the test.
The test intentionally force-kills replicas during both full sync and stable sync phases to verify the master handles these failures gracefully. The master is not supposed to crash. The SIGSEGV indicates a memory safety bug triggered by replica disconnection cleanup under concurrent write load.
### Reproduction Parameters
- **Master:** `proactor_threads=4`, `num_shards=3`
- **Replicas:** 10 total instances (`proactor_threads=2`, `num_shards=1` each)
- **Injected failures (intentional, part of test design):**
- `t_crash_fs=[2,2,2,2]` — 4 replicas are force-killed during full sync
- `t_crash_ss=[2,2,2,2]` — 4 replicas are force-killed during stable sync
- `t_disconnect=[1]` — 1 replica disconnects cleanly
- **Seeder:** `n_keys=2000`, continuous pipeline writes to the master
- **Config:** `--force_epoll=true`
### Observed Sequence
1. **21:47:19** — Master starts. 10 replicas connect, complete full sync, enter stable sync.
2. **21:47:19** — Master logs `HandleRecvSocket() error: generic:103` — expected, from intentional replica kills.
3. **21:48:30** — Master logs `OnIdle tasks are taking too long: 18026 us` — under stress.
4. **21:48:52-53** — All remaining live replicas simultaneously log `Connection reset by peer` in `STABLE_SYNC` phase — the master's TCP connection died because the master process crashed.
5. **21:48:53+** — Replicas get `Connection refused` on reconnect. `Could not shutdown socket system:107` errors during cleanup.
6. **Teardown:** `Dragonfly did not terminate gracefully, exit code -11, pid: 23093`
### Suspected Root Cause
The master SIGSEGV'd while handling a replica disconnection/cleanup event while the replication streamer was actively serializing data. Likely a stale pointer or invalid iterator dereference in the journaling or connection teardown code — triggered when multiple replicas are abruptly killed in quick succession under concurrent writes.
*(No C++ stack trace available — no core dump artifact collected by CI.)*
### Related
May be related to #7258 (`test_replication_all` SIGSEGV on the same commit), which is also a master crash during replication with concurrent writes. That crash occurs during initial RDB sync; this one occurs during stable sync. Different code paths, but same commit and same crash signal.
CI failure:
https://github.com/dragonflydb/dragonfly/actions/runs/25189859258/job/73856850478
Logs:
[logs (1).zip](https://github.com/user-attachments/files/27330836/logs.1.zip)
Contributor guide
Research direction
Start with dragonfly/replication_test.py and reproduce test_disconnect_replica with the exact parameters, commit, and --force_epoll=true setting. Review the CI logs and replication cleanup behavior around forced replica kills during stable sync; collect a crash trace if possible. Done means the master completes the test without SIGSEGV and replicas are handled cleanly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100