apache / apache/hugegraph-computer
[Task] Investigate slow integrate-test stalls in message sender barrier
- Lingua principale
- Java
- Stelle
- 65
- Fork
- 52
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Background
The `Integrate test` job for HugeGraph Computer can take a long time even though the integration suite is small. The slow part is usually not the number of tests, but a long wait in the message/input synchronization path.
An observed CI log repeatedly prints:
```text
EtcdClient - Wait for keys with prefix 'BSP_WORKER_INPUT_DONE' and timeout 86400000ms, expect 1 keys but actual got 0 keys
```
The same log shows the worker entering input step and starting vertex message sending before the wait:
```text
WorkerService inputstep started
MessageSendManager - Start sending message(type=VERTEX)
```
So the master is waiting for the worker's `BSP_WORKER_INPUT_DONE` signal, but the worker has not reached `Bsp4Worker.workerInputDone()` yet.
## Initial code pointers
- CI runs `mvn test -P integrate-test -ntp` in `.github/workflows/computer-ci.yml`.
- The `integrate-test` profile includes `IntegrateTestSuite`, which currently contains `SenderIntegrateTest`.
- `SenderIntegrateTest` has only a few cases, but `testOneWorkerWithBusyClient()` intentionally slows the send path by wrapping the client's send function with `Thread.sleep(100)`.
- `WorkerInputManager.loadGraph()` sends vertices and edges first. Only after it returns does `WorkerService.inputstep()` call `bsp4Worker.workerInputDone()`.
- `ComputerOptions.BSP_WAIT_WORKERS_TIMEOUT` and `BSP_WAIT_MASTER_TIMEOUT` default to 24 hours, so a hidden sender/session/input problem can become a very slow CI wait instead of a fast, actionable failure.
Relevant files:
```text
.github/workflows/computer-ci.yml
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/IntegrateTestSuite.java
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/SenderIntegrateTest.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/input/WorkerInputManager.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessageSender.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/MessageSendManager.java
computer/computer-api/src/main/java/org/apache/hugegraph/computer/core/config/ComputerOptions.java
```
Related prior symptom: #203 reported `The origin future must be null` in `SenderIntegrateTest`. That may be in the same control-message/future/session area, but this task is specifically about the slow CI wait and fail-fast/debuggability of the integration test.
## Suggested investigation
1. Reproduce the integration suite with etcd available:
```bash
cd computer
mvn test -P integrate-test -Dtest=IntegrateTestSuite -ntp
```
2. Confirm which test case spends time before `BSP_WORKER_INPUT_DONE`. Start with `SenderIntegrateTest#testOneWorkerWithBusyClient`.
3. Trace the input path:
```text
SenderIntegrateTest
-> WorkerService.execute()
-> WorkerService.inputstep()
-> WorkerInputManager.loadGraph()
-> MessageSendManager.startSend()/finishSend()
-> QueuedMessageSender.send()
-> Bsp4Worker.workerInputDone()
```
4. Check whether START/FINISH control futures in `QueuedMessageSender` can be left stale, completed late, or hidden behind the sender thread. The old #203 stack around `futureRef` is a useful clue.
5. Make the test fail fast and print useful diagnostics. Possible directions:
- set much smaller `bsp.wait_workers_timeout` / `bsp.wait_master_timeout` for integration tests;
- add a JUnit/test-level timeout around each integration case;
- dump worker/master thread states when the input barrier is not reached;
- ensure sender exceptions propagate to both the worker future and the master-side wait;
- replace the sleep-based busy-client simulation with a more deterministic back-pressure or blocked-client fixture.
## Expected result
- Integration tests should not spend many minutes printing only `BSP_WORKER_INPUT_DONE` wait logs.
- If the sender/input path is broken, the test should fail quickly with an actionable error and enough thread/session state to locate the failing component.
- The slow/busy-client path should have regression coverage so future changes do not reintroduce the long wait.
## Newcomer scope
This is a good newcomer task because the suspected area is narrow: one integration suite, the input-step barrier, and the message sender control future path. A complete fix does not need a large algorithm or distributed-runtime redesign; first improving timeout/diagnostics and then isolating the sender/session condition would already be valuable.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Esegui `mvn test -P integrate-test -Dtest=IntegrateTestSuite -ntp` da `computer` con etcd disponibile, quindi parti da `SenderIntegrateTest#testOneWorkerWithBusyClient`. Segui `WorkerService.inputstep()`, `WorkerInputManager.loadGraph()`, `MessageSendManager` e `QueuedMessageSender` fino a `Bsp4Worker.workerInputDone()`. Il lavoro è completato quando il percorso del client occupato ha una copertura di regressione e gli stalli falliscono rapidamente con diagnosi utilizzabili relative a sender, thread o session.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- ci-cd, distributed-systems, testing-qa
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 62/100