apache / apache/hugegraph-computer

[Task] Investigate slow integrate-test stalls in message sender barrier

Open
#350 4 comments 1 reaction 0 assignees View on GitHub
bug computer-core good first issue help wanted test
Dominant language
Java
Stars
65
Forks
52
PR merge metrics
No merged PRs in 30d

Description

## Background

The `Integrate test` job for HugeGraph Computer can take a long time even though the integration suite is small. The slow part is usually not the number of tests, but a long wait in the message/input synchronization path.

An observed CI log repeatedly prints:

```text
EtcdClient - Wait for keys with prefix 'BSP_WORKER_INPUT_DONE' and timeout 86400000ms, expect 1 keys but actual got 0 keys
```

The same log shows the worker entering input step and starting vertex message sending before the wait:

```text
WorkerService inputstep started
MessageSendManager - Start sending message(type=VERTEX)
```

So the master is waiting for the worker's `BSP_WORKER_INPUT_DONE` signal, but the worker has not reached `Bsp4Worker.workerInputDone()` yet.

## Initial code pointers

- CI runs `mvn test -P integrate-test -ntp` in `.github/workflows/computer-ci.yml`.
- The `integrate-test` profile includes `IntegrateTestSuite`, which currently contains `SenderIntegrateTest`.
- `SenderIntegrateTest` has only a few cases, but `testOneWorkerWithBusyClient()` intentionally slows the send path by wrapping the client's send function with `Thread.sleep(100)`.
- `WorkerInputManager.loadGraph()` sends vertices and edges first. Only after it returns does `WorkerService.inputstep()` call `bsp4Worker.workerInputDone()`.
- `ComputerOptions.BSP_WAIT_WORKERS_TIMEOUT` and `BSP_WAIT_MASTER_TIMEOUT` default to 24 hours, so a hidden sender/session/input problem can become a very slow CI wait instead of a fast, actionable failure.

Relevant files:

```text
.github/workflows/computer-ci.yml
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/IntegrateTestSuite.java
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/SenderIntegrateTest.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/input/WorkerInputManager.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessageSender.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/MessageSendManager.java
computer/computer-api/src/main/java/org/apache/hugegraph/computer/core/config/ComputerOptions.java
```

Related prior symptom: #203 reported `The origin future must be null` in `SenderIntegrateTest`. That may be in the same control-message/future/session area, but this task is specifically about the slow CI wait and fail-fast/debuggability of the integration test.

## Suggested investigation

1. Reproduce the integration suite with etcd available:

```bash
cd computer
mvn test -P integrate-test -Dtest=IntegrateTestSuite -ntp
```

2. Confirm which test case spends time before `BSP_WORKER_INPUT_DONE`. Start with `SenderIntegrateTest#testOneWorkerWithBusyClient`.

3. Trace the input path:

```text
SenderIntegrateTest
-> WorkerService.execute()
-> WorkerService.inputstep()
-> WorkerInputManager.loadGraph()
-> MessageSendManager.startSend()/finishSend()
-> QueuedMessageSender.send()
-> Bsp4Worker.workerInputDone()
```

4. Check whether START/FINISH control futures in `QueuedMessageSender` can be left stale, completed late, or hidden behind the sender thread. The old #203 stack around `futureRef` is a useful clue.

5. Make the test fail fast and print useful diagnostics. Possible directions:

- set much smaller `bsp.wait_workers_timeout` / `bsp.wait_master_timeout` for integration tests;
- add a JUnit/test-level timeout around each integration case;
- dump worker/master thread states when the input barrier is not reached;
- ensure sender exceptions propagate to both the worker future and the master-side wait;
- replace the sleep-based busy-client simulation with a more deterministic back-pressure or blocked-client fixture.

## Expected result

- Integration tests should not spend many minutes printing only `BSP_WORKER_INPUT_DONE` wait logs.
- If the sender/input path is broken, the test should fail quickly with an actionable error and enough thread/session state to locate the failing component.
- The slow/busy-client path should have regression coverage so future changes do not reintroduce the long wait.

## Newcomer scope

This is a good newcomer task because the suspected area is narrow: one integration suite, the input-step barrier, and the message sender control future path. A complete fix does not need a large algorithm or distributed-runtime redesign; first improving timeout/diagnostics and then isolating the sender/session condition would already be valuable.

Contributor guide

No contributing guide indexed for this repository

Research direction

Run `mvn test -P integrate-test -Dtest=IntegrateTestSuite -ntp` from `computer` with etcd available, then start at `SenderIntegrateTest#testOneWorkerWithBusyClient`. Trace `WorkerService.inputstep()`, `WorkerInputManager.loadGraph()`, `MessageSendManager`, and `QueuedMessageSender` through `Bsp4Worker.workerInputDone()`. Done means the busy-client path has regression coverage and stalls fail quickly with actionable sender, thread, or session diagnostics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ci-cd, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.