DataTalksClub / DataTalksClub/website
Restore full verification after the September mainline merges
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Outcome
Restore a deterministic, fail-closed full verification baseline on current main so release and render-impact work can receive a valid independent tester result. Diagnose the first introducing merge or shared root cause for every preserved failure, repair the product or test harness at the owning contract, and finish with green full Django plus smoke, core, and full Playwright evidence.
Normative references:
_docs/PROCESS.md_docs/ci/change-selective-ci.md_docs/ci/playwright-flake-policy.md_docs/ci/playwright-timeouts.md
Confirmed baseline and merge boundary
The preserved evidence was produced at merge commit f3f1fc66, whose parents are issue branch 51a35f60 and origin/main cec4f42e. Its tree is identical to cec4f42e (git diff cec4f42e..f3f1fc66 is empty), so these are mainline failures rather than changes introduced by #429.
The last dedicated browser-suite repair on the first-parent mainline was 71604a86 (2026-09-17). The suspect first-parent interval is therefore 71604a86..cec4f42e, especially the September 19 batch:
61130734/8cbcdaf5(public/cache and shared public-shell seams)4720c239,2128b545,704b70e7,ea57b405,fffd8177,55a7ebae,cec4f42e(account/profile migration series)e1d80a8c(shared-package pin bump)658ba5a6(wiki/docs move, including catalogue and browser-test changes)
Do not assume one merge caused every failure. Use this interval to identify the first bad tree for each root-cause cluster, and record when several node IDs reduce to one shared fixture, route, data-shape, or harness defect.
Preserved evidence on the tree-identical baseline:
- Full Django selected 4,350 tests and ended with 4,342 passed, 6 skipped, 1 expected failure, and 2 failures:
content.tests.test_catalogue_cache_binding.FailedReadRecoveryTests.test_a_failed_row_read_raises_and_the_next_read_recoversdid not raise the requiredOperationalErrorunder the normal parallel runner.core.tests.test_deployment_workflow.DeploymentWorkflowContractTests.test_gate_b_operator_contract_is_exact_and_workflow_isolateddetected a stalepyproject.tomlSHA-256 seal.
- The separately observed Playwright smoke run passed 20/30 and failed 10/30.
- The preserved full Playwright log collected 352 cases, deselected the 3 tracked-policy cases, and selected 349. It was deliberately stopped as stale shortly after 58%; before completion it recorded 174 passes, 30
FAILEDoutcomes, and 1 teardownERROR. Failures cover accessibility, account preferences, article/course/unit layout, foundation routes, historical totals, module rail, podcast graph/seasons, identity, and sponsors. This partial log is failure evidence only and cannot establish the eventual total.
Scope
- Freeze a clean post-#430
origin/mainbase and reproduce the two focused Django contracts, the maintained smoke tier, and the affected Playwright clusters before changing them. Attribute any drift from the preserved counts. - Build a failure ledger for every
FAILED/ERRORnode in the preserved log and all 10 smoke failures. For each cluster record: first bad commit or earliest bounded merge, authoritative product/spec behavior, root cause, owning fix, and final disposition. - Repair the catalogue failed-read recovery test/implementation so a failed database row read is observed, is never cached, and the immediately following read recovers under the repository's normal parallel Django runner.
- After #430 makes its reviewed
pyproject.tomlchanges, update the Gate B integrity contract through its intended seal-maintenance mechanism and prove that unauthorized drift still fails closed. - Repair each browser cluster at its owner:
- fix product code when current behavior violates the authoritative contract;
- update a stale assertion/fixture only when repository history or a current spec proves the merged behavior intentional;
- fix shared setup, isolation, teardown, server lifecycle, or data fixtures when failures are cross-test contamination or harness defects.
- Run the exact focused scenario below, then the complete maintained Django and Playwright gates with digest-bound evidence.
Non-goals
- Changing the course illustration assets or acceptance criteria in #429; #431 only restores the baseline needed to test that work.
- Reverting the September account/profile, shared-package, wiki/docs, public-shell, or data-ownership migrations as a batch.
- Treating the partial full-Playwright log as a complete failure inventory or accepting a passing prefix as evidence.
- Making failures disappear through
skip,xfail, quarantine, automatic retry, broad sleeps, timeout inflation, looser selectors/assertions, reduced viewport/route/state matrices, swallowed exceptions, or a smaller smoke/core/full selection. - Disabling Django parallelism globally or forcing the entire suite serial to hide cache/process isolation defects.
- Refreshing a frozen hash without reviewing and documenting the file change it approves.
- Fixing unrelated product behavior discovered after the implementation base is frozen; new drift must be attributed and split only when it is independently verifiable and does not leave this gate red.
Dependencies and ordering
- #430 must land first. It intentionally changes
pyproject.toml, while the failing Gate B contract seals that exact file. #431 must review and seal the post-#430 content once, avoiding conflicting or immediately stale hashes. - #429 is blocked by #431. Its tree was not responsible for this baseline, but its render-impact tester gate cannot pass while main's required browser suites are red.
- The browser and catalogue investigations may proceed in parallel in isolated worktrees, but their fixes must be integrated onto the same frozen post-#430 base and receive one complete final verification run.
community-baseis shared with AI Shipping Labs. If diagnosis requires a package change, followAGENTS.mdand run both consuming projects' suites; a local fixture/adapter repair does not authorize a package change by itself.
Acceptance criteria
- The handoff records a clean, exact post-#430
origin/mainbase SHA and a ledger accounting for both Django failures, all 10 observed smoke failures, and everyFAILED/ERRORnode in the preserved full Playwright log. Repeated viewport failures may share one root cause, but no node is silently omitted. - Each ledger cluster names the first bad commit or narrowest proven merge interval in
71604a86..cec4f42e(or explains with evidence why the defect predates that boundary), the authoritative intended behavior, and whether the fix belongs to product code, test expectation/fixture, or harness isolation. - The catalogue recovery contract deterministically observes one
OperationalError, does not cache the failed result, and returns published courses on the next read undermanage.py test --parallel; it passes at least three consecutive focused runs without serializing the repository suite. - The Gate B integrity test approves the reviewed post-#430
pyproject.tomlcontent and still fails when that file is synthetically modified without updating the seal. The change does not weaken or remove any other frozen-file assertion. -
uv run --frozen python scripts/ci.py test-django-fullexits zero with a complete terminal summary. No previously selected test is removed, skipped, or converted to an expected failure to obtain that result. -
uv run --frozen python scripts/ci.py test-playwright-smokecompletes 30/30 with zero failures/errors, zero reruns, and no newly quarantined cases. -
uv run --frozen python scripts/ci.py test-playwright-coreexits zero with complete policy/evidence summaries and unchanged profile semantics. -
uv run --frozen python scripts/ci.py test-playwrightcompletes the entire selected set with zero failures/errors and zero reruns. The three existing policy deselections may remain only if still selected by the tracked quarantine policy; report the final collected/selected/deselected counts rather than hard-coding 349 if main legitimately added tests. - The desktop and mobile accessibility registry cases complete inside the existing diagnostic/runner budgets; no 120-second diagnostic is reclassified as success and no timeout is increased merely to hide a deadlock.
- Account theme precedence/first-paint/storage-denial/save-failure/settings navigation, public hub/docs/FAQ redirects, article/course/unit measure, module-rail collapse, podcast graph/native fallback/card keyboard behavior, identity navigation, historical-total Studio flow, and sponsor management all retain their asserted user-visible and safe-denial contracts at the viewports represented in the tests.
- No fix adds
skip,xfail, quarantine, retry, broad sleep, global serialization, reduced test/viewport coverage, relaxed redirect/status/accessibility assertions, or exception swallowing. Any expectation change cites the spec or intentional merge that makes the old expectation stale. - The final verification plan and evidence are generated from the same frozen base/head, independently recomputed by the tester, and contain terminal complete results. If product/templates/render inputs change, the graph-selected desktop/mobile screenshots are captured and inspected; if the repair is test/harness-only, screenshots are explicitly
not_applicablewith path-based justification.
Exact focused verification scenario
Run the two Django contracts together under the same parallel runner shape three consecutive times:
for run in 1 2 3; do
uv run --frozen python manage.py test --parallel --noinput \
content.tests.test_catalogue_cache_binding.FailedReadRecoveryTests.test_a_failed_row_read_raises_and_the_next_read_recovers \
core.tests.test_deployment_workflow.DeploymentWorkflowContractTests.test_gate_b_operator_contract_is_exact_and_workflow_isolated \
|| exit 1
done
Prove the seal remains fail-closed with a temporary copy or dedicated unit fixture that changes pyproject.toml bytes without changing the expected digest; do not mutate the candidate worktree's real file as evidence.
Then run the maintained browser tiers in escalation order:
uv run --frozen python scripts/ci.py test-playwright-smoke
uv run --frozen python scripts/ci.py test-playwright-core
uv run --frozen python scripts/ci.py test-playwright
The focused browser review must explicitly inspect the failure-ledger clusters at their original desktop/mobile, JavaScript-on/off, keyboard, storage-denial, and safe-redirect states. Finish with:
uv run --frozen python scripts/ci.py test-django-full
Expected result: complete green terminal summaries for full Django and all three blocking Playwright profiles, with policy counters internally consistent (attempted = passed + failed + skipped, failed = 0, rerun = 0) and no required evidence disposition left pending or skipped.
Verification handoff
- Preserve the failure ledger, exact commands, base/head SHAs, exit codes, counts, and evidence digests below
.tmp/. - Report the first-bad history result separately from the fix commit so reviewers can assess whether an expectation update is legitimate.
- Leave the integrated candidate frozen and uncommitted for an independent tester. The tester reruns the maintained commands; implementation-agent output alone is not acceptance.
- PM acceptance occurs only after the independent tester passes every criterion. Commit afterward with
Closes #431using the repository's local no-PR merge process.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with _docs/PROCESS.md and the linked CI policies, then reproduce the two named Django contracts three times using the exact parallel command. Review the preserved Playwright smoke and full-suite evidence, build the required failure ledger, and trace each cluster through the listed merge interval. Done means the focused contracts, full Django suite, and smoke, core, and full Playwright commands complete with the specified green, no-rerun evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, python
- Domain
- backend, frontend, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100