DataTalksClub / DataTalksClub/website

test-playwright-smoke fails on its own 600s cap with zero test failures

Open Beginner friendly
#329 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug infra operations P1 testing
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

make test-playwright-smoke wraps its pytest selection in a hardcoded
timeout … 600s (Makefile:311). On this development host the tier needs
roughly 675s, so the target exits 124 with zero test failures, at 66-93%
through a run in which every executed test passed.

Four independent observations across four different worktrees on 2026-09-07:

Lane Invocation Result
#323 engineer capped Error 124, 0 failed
#323 tester capped Error 124, 600.3s, 28 passed, 0 failed
#323 tester uncapped, identical selection 30 passed, exit 0, 484.68s
#323 tester capped, under load Error 124, 600.5s, 0 failed, 73%
#324 engineer capped, at the pristine base 6899c616 Error 124
#324 tester capped at base / capped at head / uncapped 124 / 124 / 30 passed in 899.76s
#322 engineer capped 30 tests printed PASSED to 100%, then Error 124

Two properties make this a real defect rather than a slow machine:

  1. It fails identically at the pristine base, so it is not attributable to
    any change under review. Every lane that hit it had to spend time proving
    that separately.
  2. It is marginal, not hopeless. The fastest uncapped run of the identical
    selection was 484s against a 600s cap — about 20% headroom on a quiet box
    and none under load. So it passes or fails depending on what else is running,
    which is the worst of both worlds: it cannot be trusted as a signal and it
    cannot be dismissed as always-red.

test-playwright-smoke is the only tier carrying a hardcoded cap
(_docs/runbooks/Makefile grep). The consequence is that the tier every
backend-only change is required to run is the one tier that cannot be relied on
to report its own result, and every reviewer pays the cost of re-establishing
that independently.

What to decide: raise the cap, scale it to the host, or remove it and let the
suite's own per-test timeouts govern. #252 covers the full tier's one-hour
bound and is a different bound on a different tier; this is not that.

Not in scope: making the smoke tier faster.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Makefile:311 and inspect the command used by test-playwright-smoke, then compare its capped behavior with the identical uncapped pytest selection described in the issue. Confirm the chosen cap strategy against the existing runbook references and verify that the target completes with the smoke tests' actual result instead of a false timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.