stacklok / stacklok/mecatl

Latency-injecting gRPC test harness for race-prone flows (steer, handoff, queue drain)

Open
#637 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Problem

Concurrency/race-heavy surfaces (steer-while-running #512 / PR #570, the active-run handoff, promote-grace, supersede/append, queue drain) are tested over an in-process bufconn with ~zero gRPC latency. The interesting interleavings (ack vs echo reordering, drain-vs-append races, terminal-race promote) all depend on timing the local harness never produces. Real deployments (TUI → remote mecated, k8s) have RTTs that make these races common.

Ask

A latency-injecting test harness for the gRPC transport so race-prone flows can be exercised under realistic delay:

  • A wrapper (listener or stream interceptor) that adds configurable latency / jitter to Converse stream frames, or
  • A test-only dial option to slow the bufconn path, so existing tests can be re-run under latency (a -race-style latency matrix), or
  • Targeted fault-injection points (delay acks vs echoes independently) for the steer ack/echo correlation and the active-run handoff.

Goal: reproduce the class of bugs that only appear when client and server are separated by a real network (e.g. the append-vs-drain ordering, ack-after-close, promote-grace timing) deterministically in CI.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing in-process bufconn tests for steer, active-run handoff, promote-grace, supersede/append, and queue drain. Compare latency or jitter injection options, then verify that the chosen harness reproduces ordering races deterministically in CI, including ack-versus-echo, append-versus-drain, ack-after-close, and promote-grace timing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.