rpc-tests.yml is chronically flaky on main

Open
#2,689 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
github-actions, rust
Domain
ci-cd, testing-qa

Research direction

Start with rpc-tests.yml and compare the listed failed workflow runs, including the clawback and auto-build integration tests, to identify recurring timing or resource patterns. Review the integration test execution and cargo-nextest retry options mentioned in the issue. Done should be a bounded workflow or test-suite change that reduces these failures without leaving main gated by unresolved flakiness.

Written by the indexing model from the issue text.

Description

The RPC Tests workflow fails on a large majority of pushes to main: 23 of the last 30 push-triggered runs failed. The failures don't point to one regression — a different integration test fails each time, and the failing job varies too (seen on both ubuntu-latest-8-cores and macos-15-large). For example, integration::tx::clawback::clawback has failed on an immediate post-clawback balance assertion on macOS, while integration::auto_build::deploy_workspace_with_package_auto_builds has failed on Ubuntu in a different run. This pattern looks like timing/eventual-consistency races and possibly resource contention in the live-network integration suite rather than a single code defect.

A few directions that might be worth exploring: replacing immediate post-action assertions (like the clawback balance check) with a poll-with-timeout, adding retries for the integration test suite (e.g. via cargo-nextest --retries), and possibly separating out the flakiest tests into a non-blocking job so they stop gating main while they're investigated.

Below are 15 recent runs on main where this workflow failed, most recent first, as evidence of the pattern:

Dominant language
Rust
Stars
123
Forks
141
Avg merge
2d 21h
Merged PRs (30d)
17

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stellar/stellar-cli

All issues in stellar/stellar-cli

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.