apache / apache/pekko

Flaky test: RemotingSpec "should resume the outbound reader when passive read handoff fails" on Scala 3

Open
#3,308 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
211
Avg merge
1d 6h
Merged PRs (30d)
89

Description

### Description

The test `RemotingSpec` — `"should resume the outbound reader when passive read handoff fails"` intermittently times out on the Scala 3 CI job.

### Failure

```
java.lang.AssertionError: assertion failed: timeout (6 seconds) during expectMsg while waiting for after-handoff-failure
at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
at org.apache.pekko.testkit.TestKitBase.expectMsg_internal(TestKit.scala:471)
at org.apache.pekko.testkit.TestKitBase.expectMsg(TestKit.scala:448)
at org.apache.pekko.remote.classic.RemotingSpec.f$proxy19$1(RemotingSpec.scala:797)
```

### Location

`remote/src/test/scala/org/apache/pekko/remote/classic/RemotingSpec.scala:797`

```scala
outboundRemoteHandle.association.write(payload("after-handoff-failure"))
receiverProbe.expectMsg("after-handoff-failure") // <-- times out here
```

### Observed in

- PR #3095 (unrelated changes — `Source.queue` deprecation): [CI run](https://github.com/apache/pekko/actions/runs/28965711798/job/85948213045)
- The test was added by PR #3205

### Analysis

The test sets up a passive read-only inbound association, sends a broken packet to trigger disassociation of the read-only endpoint, then expects the outbound writer to resume delivering messages. The `expectMsg` at line 797 times out, suggesting the outbound reader did not resume after the passive read handoff failure.

This is a timing-sensitive test involving multiple transport associations and handoffs. The 6-second timeout may be insufficient under CI load, or the resume logic may have a race condition that manifests more often on Scala 3 (possibly due to different compilation/optimization characteristics affecting thread scheduling).

### Possible fixes

1. Increase the `expectMsg` timeout with a `dilated` factor
2. Investigate whether the resume logic in the endpoint reader has a race condition
3. Add retry or `awaitAssert` around the assertion to tolerate transient timing

Contributor guide

Open the contributing guide

Research direction

Start with remote/src/test/scala/org/apache/pekko/remote/classic/RemotingSpec.scala around line 797 and reproduce the named test on the Scala 3 CI job. Trace the passive read handoff and outbound reader behavior after the broken packet, then determine whether the timeout or resume logic is responsible. Done means the test reliably delivers "after-handoff-failure" without masking a real failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.