dragonflydb / dragonflydb/dragonfly

test_disconnect_replica failed: master crashes with SIGSEGV when replicas are force-killed during stable sync

Open
#7,259 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

### Environment
* **Test:** `test_disconnect_replica[df_factory0-df_seeder_factory0-4-t_crash_fs1-t_crash_ss1-t_disonnect1-2000]`
* **File:** `dragonfly/replication_test.py`
* **Environment:** CI (epoll regression tests, `--force_epoll=true`, debug build)
* **Commit:** `01e8e60e0cad7da29eba6ff3517a177bf19ab764`

### Executive Summary
The **master** Dragonfly process (pid `23093`, port `40015`) crashed with **SIGSEGV (exit code -11)** approximately 93 seconds into the test.

The test intentionally force-kills replicas during both full sync and stable sync phases to verify the master handles these failures gracefully. The master is not supposed to crash. The SIGSEGV indicates a memory safety bug triggered by replica disconnection cleanup under concurrent write load.

### Reproduction Parameters
- **Master:** `proactor_threads=4`, `num_shards=3`
- **Replicas:** 10 total instances (`proactor_threads=2`, `num_shards=1` each)
- **Injected failures (intentional, part of test design):**
- `t_crash_fs=[2,2,2,2]` — 4 replicas are force-killed during full sync
- `t_crash_ss=[2,2,2,2]` — 4 replicas are force-killed during stable sync
- `t_disconnect=[1]` — 1 replica disconnects cleanly
- **Seeder:** `n_keys=2000`, continuous pipeline writes to the master
- **Config:** `--force_epoll=true`

### Observed Sequence
1. **21:47:19** — Master starts. 10 replicas connect, complete full sync, enter stable sync.
2. **21:47:19** — Master logs `HandleRecvSocket() error: generic:103` — expected, from intentional replica kills.
3. **21:48:30** — Master logs `OnIdle tasks are taking too long: 18026 us` — under stress.
4. **21:48:52-53** — All remaining live replicas simultaneously log `Connection reset by peer` in `STABLE_SYNC` phase — the master's TCP connection died because the master process crashed.
5. **21:48:53+** — Replicas get `Connection refused` on reconnect. `Could not shutdown socket system:107` errors during cleanup.
6. **Teardown:** `Dragonfly did not terminate gracefully, exit code -11, pid: 23093`

### Suspected Root Cause
The master SIGSEGV'd while handling a replica disconnection/cleanup event while the replication streamer was actively serializing data. Likely a stale pointer or invalid iterator dereference in the journaling or connection teardown code — triggered when multiple replicas are abruptly killed in quick succession under concurrent writes.

*(No C++ stack trace available — no core dump artifact collected by CI.)*

### Related
May be related to #7258 (`test_replication_all` SIGSEGV on the same commit), which is also a master crash during replication with concurrent writes. That crash occurs during initial RDB sync; this one occurs during stable sync. Different code paths, but same commit and same crash signal.

CI failure:
https://github.com/dragonflydb/dragonfly/actions/runs/25189859258/job/73856850478
Logs:

[logs (1).zip](https://github.com/user-attachments/files/27330836/logs.1.zip)

Contributor guide

Open the contributing guide

Research direction

Start with dragonfly/replication_test.py and reproduce test_disconnect_replica with the exact parameters, commit, and --force_epoll=true setting. Review the CI logs and replication cleanup behavior around forced replica kills during stable sync; collect a crash trace if possible. Done means the master completes the test without SIGSEGV and replicas are handled cleanly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.