stacklok / stacklok/mecatl

Flaky test: TestSteer_PromotedRelaySequential still races terminal steer closure

Open Beginner friendly
#815 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

flake
Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Summary

internal/adapter/server/steer_handoff_test.go's TestSteer_PromotedRelaySequential (R3-3 in that file's header) intermittently fails under -race/CI load with:

--- FAIL: TestSteer_PromotedRelaySequential (0.11s)
    steer_handoff_test.go:342: results on the stream = 1 ([promoted follow-up]), want 2 (original + promoted follow-up): [steer.outcome steer turn.start message.delta turn.end result]

Observed failing on main (run 32901961661, 2026-08-25) and again on PR #796 (run 33150944649, 2026-08-28) — same assertion, same log shape both times.

Root cause

The test held the original run "terminal-but-registered" via a gRPC send-stall, sent a late steer, then used a fixed time.Sleep(100 * time.Millisecond) before releasing the stall, assuming that was enough time for the server to classify the steer as too_late and promote a follow-up run.

That sleep doesn't synchronize with anything real: the server's internal closeSteerDrained transition (engine/agent/steer.go, called from terminate* in loop.go) happens synchronously on the engine goroutine right after EvTurnEnd, independent of wall-clock time. Under CI scheduling contention, the steer can occasionally still be classified as live-accepted instead of too-late, merging it into the original run instead of promoting a new one — exactly the observed symptom (only one result on the stream, from the second mock turn).

This is the same bug class as #676 (TestSteer_AwaitingResumeDrains, closed) and the one fixed for TestSteer_ControlTargetsPromotedRun in #672 — that fix (poll svc.LookupRun until the registered run pointer changes from the captured original, instead of guessing at timing) was never applied to this sibling test.

Fix

Apply the same poll-based synchronization from #672 to TestSteer_PromotedRelaySequential: capture the original run pointer via svc.LookupRun before sending the steer, then poll (bounded by the test's own context deadline) until a different pointer is registered, before releasing the stall.

Fix branch: fix/steer-promoted-relay-flake.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/adapter/server/steer_handoff_test.go at TestSteer_PromotedRelaySequential, then compare the poll-based synchronization used by TestSteer_ControlTargetsPromotedRun in #672. Run the test with -race and use the test context deadline while checking svc.LookupRun before releasing the send stall. Done means the test reliably observes the promoted run and passes under CI-like race conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.