CivicDataLab / CivicDataLab/ParakhAI_test

E2E shards report success despite dozens of failing tests; one test file is never collected

Open
#12 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2h 10m
Merged PRs (30d)
1

Description

Two separate gaps found while adding the Keycloak migration tests (#11).

1. Shards report success despite failing tests

On the PR run for #11:

E2E Tests (shard 2/3): SUCCESS   -> 26 failed, 85 passed, 13 errors
E2E Tests (shard 3/3): SUCCESS   -> 24 failed, 44 passed, 10 errors

The GitHub check is green while 50 tests failed. Whatever decides the job's exit status is not reflecting the pytest result, so nobody sees these.

This is the same class of problem as CivicDataSpace-test#22 — a green CI signal that asserts nothing — and it means this suite's green cannot currently be trusted as evidence.

2. test_add_model_flow.py is collected by no shard

Grepping all three shard logs for add_model returns 0 matches, so the file never runs in CI. The session-expiry guard repaired in #11 therefore has no CI coverage; it was verified directly against the live URL shapes instead.

Worth checking whether other files are silently uncollected by the sharding.

Also observed

dev.api.parakh.civicdataspace.in returned ReadTimeout on several API tests during these runs. Checked for the same unconditional user.save() pattern that caused DataSpaceBackend#134 and found none, so this is a different cause — noted here rather than assumed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the E2E shard workflow and the shard logs from PR #11, then trace how the pytest result becomes the GitHub job exit status. Check why test_add_model_flow.py is absent from all three shard logs and whether other files are also uncollected; done means failing tests make the check fail and every intended test file is assigned to a shard.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.