dragonflydb / dragonflydb/dragonfly
test_migration_timeout_on_sync: CI timeout due to costly re-sync after intentional migration failure
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
#### Environment
* **Test:** `dragonfly/cluster_test.py::test_migration_timeout_on_sync`
* **Commit:** `de7d1f50a4d8648f3a79de89b21865ddc74940e6`
* **CI Link:** [https://github.com/dragonflydb/dragonfly/actions/runs/25295832950](https://github.com/dragonflydb/dragonfly/actions/runs/25295832950)
---
#### What Happens
The test intentionally triggers a `JournalStreamer write operation timeout` by pausing the target node mid-migration. While the engine correctly attempts to auto-retry the migration to maintain consistency, the retry requires a full data re-sync. Given the test's throttled CPU budget (`migration_buckets_cpu_budget=1`), this re-sync is too slow to complete within the 300s `pytest-timeout` window.
#### Evidence from Logs (Authentic Traces)
**1. Intentional timeout fires on Master (PID 17600):**
```text
W20260504 01:08:40.785347 17607 streamer.cc:373] Stream timed out, inflight bytes/sent start: 157269/21719995, end: 157269/21719995
W20260504 01:08:40.785640 17607 execution_state.cc:93] ReportError: JournalStreamer write operation timeout
W20260504 01:08:40.789935 17607 outgoing_slot_migration.cc:166] Finish outgoing migration for 047df6... with error: JournalStreamer write operation timeout
```
**2. Target (PID 17601) detects disconnection and cancels incoming flows:**
```text
W20260504 01:08:40.890379 17606 incoming_slot_migration.cc:83] Error reading from migration socket for shard 2: Input/output error, socket state: State: CLOSE_WAIT...
I20260504 01:08:40.905144 17604 incoming_slot_migration.cc:135] Flow 2 canceled
```
**3. On retry, the Target flushes slots and the Source reconnects to start over:**
```text
I20260504 01:08:41.290551 17601 cluster_family.cc:982] Flushing slots during migration reinitialization 047df6..., slots: [0, 16383]
I20260504 01:08:41.295787 17602 outgoing_slot_migration.cc:58] Connecting to source node_id 047df6... shard_id 1
```
**4. The test is killed exactly at the 300s mark by the pytest watchdog:**
```text
E Failed: Timeout (>300.0s) from pytest-timeout.
...
FAILED 😰 dragonfly/cluster_test.py::test_migration_timeout_on_sync[df_seeder_factory0-df_factory0] - Failed: Timeout (>300.0s) from pytest-timeout.
```
#### Root Cause
The test populates the master with **300,000 keys** (~300MB). [cite_start]When the migration retries, the target node must flush all partially migrated data to ensure a clean state[cite: 50]. Re-transferring 300MB with `migration_buckets_cpu_budget=1` on a loaded CI runner under the load of a background seeder exceeds the remaining ~293 seconds of the test budget.
#### Suggested Fix
The most efficient solution is to **reduce the dataset size**. A target of 50,000 keys is more than enough to saturate the socket buffers and trigger the intentional timeout, but it makes the subsequent re-sync 6x faster.
**Modify `tests/dragonfly/cluster_test.py`:**
```python
# Before
await DebugPopulateSeeder(key_target=300000, data_size=1000).run(nodes[0].client)
# After
await DebugPopulateSeeder(key_target=50000, data_size=1000).run(nodes[0].client)
```
Contributor guide
Research direction
Start in tests/dragonfly/cluster_test.py at dragonfly/cluster_test.py::test_migration_timeout_on_sync and review the DebugPopulateSeeder setup and migration timeout configuration. Run the parametrized test before and after the dataset adjustment; done means the intentional timeout still occurs while the test completes within the 300-second pytest-timeout window.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100