DataTalksClub / DataTalksClub/website
test-playwright-smoke fails on its own 600s cap with zero test failures
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
make test-playwright-smoke wraps its pytest selection in a hardcoded
timeout … 600s (Makefile:311). On this development host the tier needs
roughly 675s, so the target exits 124 with zero test failures, at 66-93%
through a run in which every executed test passed.
Four independent observations across four different worktrees on 2026-09-07:
| Lane | Invocation | Result |
|---|---|---|
| #323 engineer | capped | Error 124, 0 failed |
| #323 tester | capped | Error 124, 600.3s, 28 passed, 0 failed |
| #323 tester | uncapped, identical selection | 30 passed, exit 0, 484.68s |
| #323 tester | capped, under load | Error 124, 600.5s, 0 failed, 73% |
| #324 engineer | capped, at the pristine base 6899c616 |
Error 124 |
| #324 tester | capped at base / capped at head / uncapped | 124 / 124 / 30 passed in 899.76s |
| #322 engineer | capped | 30 tests printed PASSED to 100%, then Error 124 |
Two properties make this a real defect rather than a slow machine:
- It fails identically at the pristine base, so it is not attributable to
any change under review. Every lane that hit it had to spend time proving
that separately. - It is marginal, not hopeless. The fastest uncapped run of the identical
selection was 484s against a 600s cap — about 20% headroom on a quiet box
and none under load. So it passes or fails depending on what else is running,
which is the worst of both worlds: it cannot be trusted as a signal and it
cannot be dismissed as always-red.
test-playwright-smoke is the only tier carrying a hardcoded cap
(_docs/runbooks/Makefile grep). The consequence is that the tier every
backend-only change is required to run is the one tier that cannot be relied on
to report its own result, and every reviewer pays the cost of re-establishing
that independently.
What to decide: raise the cap, scale it to the host, or remove it and let the
suite's own per-test timeouts govern. #252 covers the full tier's one-hour
bound and is a different bound on a different tier; this is not that.
Not in scope: making the smoke tier faster.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Makefile:311 and inspect the command used by test-playwright-smoke, then compare its capped behavior with the identical uncapped pytest selection described in the issue. Confirm the chosen cap strategy against the existing runbook references and verify that the target completes with the smoke tests' actual result instead of a false timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100