dragonflydb / dragonflydb/dragonfly

test_migration_timeout_on_sync: CI timeout due to costly re-sync after intentional migration failure

Open Beginner friendly
#7,261 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

#### Environment
* **Test:** `dragonfly/cluster_test.py::test_migration_timeout_on_sync`
* **Commit:** `de7d1f50a4d8648f3a79de89b21865ddc74940e6`
* **CI Link:** [https://github.com/dragonflydb/dragonfly/actions/runs/25295832950](https://github.com/dragonflydb/dragonfly/actions/runs/25295832950)

---

#### What Happens
The test intentionally triggers a `JournalStreamer write operation timeout` by pausing the target node mid-migration. While the engine correctly attempts to auto-retry the migration to maintain consistency, the retry requires a full data re-sync. Given the test's throttled CPU budget (`migration_buckets_cpu_budget=1`), this re-sync is too slow to complete within the 300s `pytest-timeout` window.

#### Evidence from Logs (Authentic Traces)

**1. Intentional timeout fires on Master (PID 17600):**
```text
W20260504 01:08:40.785347 17607 streamer.cc:373] Stream timed out, inflight bytes/sent start: 157269/21719995, end: 157269/21719995
W20260504 01:08:40.785640 17607 execution_state.cc:93] ReportError: JournalStreamer write operation timeout
W20260504 01:08:40.789935 17607 outgoing_slot_migration.cc:166] Finish outgoing migration for 047df6... with error: JournalStreamer write operation timeout
```

**2. Target (PID 17601) detects disconnection and cancels incoming flows:**
```text
W20260504 01:08:40.890379 17606 incoming_slot_migration.cc:83] Error reading from migration socket for shard 2: Input/output error, socket state: State: CLOSE_WAIT...
I20260504 01:08:40.905144 17604 incoming_slot_migration.cc:135] Flow 2 canceled
```

**3. On retry, the Target flushes slots and the Source reconnects to start over:**
```text
I20260504 01:08:41.290551 17601 cluster_family.cc:982] Flushing slots during migration reinitialization 047df6..., slots: [0, 16383]
I20260504 01:08:41.295787 17602 outgoing_slot_migration.cc:58] Connecting to source node_id 047df6... shard_id 1
```

**4. The test is killed exactly at the 300s mark by the pytest watchdog:**
```text
E Failed: Timeout (>300.0s) from pytest-timeout.
...
FAILED 😰 dragonfly/cluster_test.py::test_migration_timeout_on_sync[df_seeder_factory0-df_factory0] - Failed: Timeout (>300.0s) from pytest-timeout.
```

#### Root Cause
The test populates the master with **300,000 keys** (~300MB). [cite_start]When the migration retries, the target node must flush all partially migrated data to ensure a clean state[cite: 50]. Re-transferring 300MB with `migration_buckets_cpu_budget=1` on a loaded CI runner under the load of a background seeder exceeds the remaining ~293 seconds of the test budget.

#### Suggested Fix
The most efficient solution is to **reduce the dataset size**. A target of 50,000 keys is more than enough to saturate the socket buffers and trigger the intentional timeout, but it makes the subsequent re-sync 6x faster.

**Modify `tests/dragonfly/cluster_test.py`:**
```python
# Before
await DebugPopulateSeeder(key_target=300000, data_size=1000).run(nodes[0].client)

# After
await DebugPopulateSeeder(key_target=50000, data_size=1000).run(nodes[0].client)
```

Contributor guide

Open the contributing guide

Research direction

Start in tests/dragonfly/cluster_test.py at dragonfly/cluster_test.py::test_migration_timeout_on_sync and review the DebugPopulateSeeder setup and migration timeout configuration. Run the parametrized test before and after the dataset adjustment; done means the intentional timeout still occurs while the test completes within the 300-second pytest-timeout window.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
testing
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.