CivicDataLab / CivicDataLab/ParakhAI_test
E2E shards report success despite dozens of failing tests; one test file is never collected
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2h 10m
- Merged PRs (30d)
- 1
Description
Two separate gaps found while adding the Keycloak migration tests (#11).
1. Shards report success despite failing tests
On the PR run for #11:
E2E Tests (shard 2/3): SUCCESS -> 26 failed, 85 passed, 13 errors
E2E Tests (shard 3/3): SUCCESS -> 24 failed, 44 passed, 10 errors
The GitHub check is green while 50 tests failed. Whatever decides the job's exit status is not reflecting the pytest result, so nobody sees these.
This is the same class of problem as CivicDataSpace-test#22 — a green CI signal that asserts nothing — and it means this suite's green cannot currently be trusted as evidence.
2. test_add_model_flow.py is collected by no shard
Grepping all three shard logs for add_model returns 0 matches, so the file never runs in CI. The session-expiry guard repaired in #11 therefore has no CI coverage; it was verified directly against the live URL shapes instead.
Worth checking whether other files are silently uncollected by the sharding.
Also observed
dev.api.parakh.civicdataspace.in returned ReadTimeout on several API tests during these runs. Checked for the same unconditional user.save() pattern that caused DataSpaceBackend#134 and found none, so this is a different cause — noted here rather than assumed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the E2E shard workflow and the shard logs from PR #11, then trace how the pytest result becomes the GitHub job exit status. Check why test_add_model_flow.py is absent from all three shard logs and whether other files are also uncollected; done means failing tests make the check fail and every intended test file is assigned to a shard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100