Skyvern-AI / Skyvern-AI/rustwright

CI: `cargo and fast pytest` flakes on navigation timeouts (~30% of main runs)

Open
#211 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
890
Forks
58
Avg merge
20h 29m
Merged PRs (30d)
12

Description

Summary

The cargo and fast pytest job in test.yml fails intermittently on main with navigation timeouts. Over the last 10 runs on main, 3 failed — all three in this same job, none reproducible locally, and each with a different commit underneath. The failures are timeouts waiting for a navigation to settle, not assertion failures, so they look like an environment/timing problem rather than a product regression.

This is currently the main source of red CI on main, and it makes a genuinely-broken branch hard to distinguish from a flake.

Observed failures

Python side (2 of 3), tests/test_rustwright_sync_api.py::test_page_route_fulfill_can_override_fetched_response:

FAILED tests/test_rustwright_sync_api.py::test_page_route_fulfill_can_override_fetched_response
  - rustwright.sync_api.TimeoutError: Page.goto: Timeout 30000ms exceeded.
========== 1 failed, 68 passed, 1205 deselected in 122.02s ==========

Rust side (1 of 3), actor::tests::password_snapshot_masks_same_document_root_replacement_echo:

thread 'actor::tests::password_snapshot_masks_same_document_root_replacement_echo' panicked at src/actor.rs:9102:14:
  navigate actor to root-replacement password echo fixture: Timeout(10000)
test result: FAILED. 85 passed; 1 failed

Common shape

Both are goto/navigate waits against a locally-served fixture that time out under load, in a job that runs the Rust suite and the fast Python subset back to back on one runner. The Python case is the more frequent one and it is a route-interception test, so the navigation depends on the route handler being installed and the fulfill path completing.

Suggested next steps

  • Confirm whether the fixture server is fully ready before the navigation starts, rather than relying on the navigation timeout to absorb startup latency.
  • Check whether running the Rust suite and the Python subset in the same job starves the browser of CPU; splitting them, or serialising browser-backed tests, would tell us quickly.
  • If the timeouts are genuinely load-related, raise the navigation budget for these fixtures specifically rather than globally, so a real hang still fails fast.

Why this matters now

CI red-vs-green is currently not trustworthy on main. Any work that needs a clean baseline has to re-run the job and eyeball which test failed, which is exactly the kind of manual triage that hides a real regression.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cargo and fast pytest job in test.yml, then inspect tests/test_rustwright_sync_api.py::test_page_route_fulfill_can_override_fetched_response and the Rust test at src/actor.rs:9102. Re-run the job and check fixture readiness and resource contention between the Rust and Python suites. Done means the navigation flake is identified and CI reliably distinguishes real failures from timing-related failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python, rust
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.