openwisp / openwisp/docker-openwisp
[tests] Stabilize flaky integration tests and reorganize the suite
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 209
- Forks
- 125
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 24
Description
Is your feature request related to a problem? Please describe.
tests/runtests.py contains 1,279 lines across preconditions, dashboard configuration checks, local utility checks, OpenVPN coverage, and 18 functional Selenium tests. Its single TestServices class owns application setup, persistent browser sessions, test fixtures, and unrelated feature checks, which makes it difficult to identify and fix state leaks.
An analysis of the latest 100 CI Build (Docker-OpenWISP) workflow runs from 2026-07-23 through 2026-08-31 found that 89 runs reached the Test step and executed 115 complete Python test-suite attempts. Twenty-eight attempts failed. The CI retry wrapper retried 26 of them, and 10 workflow runs eventually reported success despite one or more failed suite attempts. The retry loop currently reuses the same Compose stack, database, Redis state, volumes, and browser-related test state, so a green workflow does not establish that its first test attempt was clean.
The recurring failures were:
| Test | Failed results | Affected CI runs | Failure signature |
|---|---|---|---|
test_add_superuser |
12 | 8 | StaleElementReferenceException |
test_forgot_password |
10 | 7 | Password-reset confirmation was absent |
test_console_errors |
6 | 4 | Five stale-element errors and one failed static asset |
test_redis_buckets_are_separated |
5 | 1 | Deterministic Redis URL parsing assertion |
test_topology_graph |
3 | 1 | Deterministic topology rendering assertion |
The last two rows were deterministic regressions that exhausted all five retries. The Redis parsing issue has since been corrected. The former topology implementation has been replaced by the deterministic database-backed fixture introduced in #682. The first three rows are recurring Selenium synchronization failures that retries conceal.
Describe the solution you'd like
The primary goal is to stabilize the recurring Selenium failures so CI retries no longer conceal them. Reorganize the suite as supporting cleanup: it makes the browser, local utility, and container checks easier to navigate while preserving the existing test lifecycle and targeted-test entry point.
Split the existing test classes into a few direct files while keeping their code and lifecycle intact. Keep tests/runtests.py as a small compatibility runner that imports and re-exports the classes, so make runtests, make develop-pythontests, and python3 tests/runtests.py TestServices.test_name keep working.
tests/
runtests.py # Existing entry point and imports
test_core.py # Preconditions and dashboard configuration checks
test_selenium.py # TestServices unchanged
test_local_utils.py # TestLocalUtils unchanged
test_openvpn.py # TestOpenVPN unchanged
utils.py # Existing shared helpers
The reorganization alone does not isolate state leaks: TestServices will retain its existing class-level setup, persistent browser sessions, and test order. Do not introduce new test framework layers, mixins, fixture abstractions, or a new discovery command. Move each existing class as a whole and retain its current imports, setup, cleanup, and test names.
Address the recurring failures in focused follow-up PRs:
- Make user creation and resource selection wait for a fresh clickable element, avoiding the recurring stale-element failures.
- Make
test_forgot_passwordwait for its explicit success state instead of inspecting the page source after generic readiness. - Keep the CI retry wrapper until these failures are fixed, but ensure its logs continue to show every failed suite attempt.
Acceptance criteria:
python3 tests/runtests.py TestServices.test_add_superusercontinues to resolve and run the named test.make develop-pythontestscontinues to execute all 35 tests throughtests/runtests.py.- The reorganization preserves class import order, test names, setup, cleanup, and all
__file__-relative test paths. - The focused Selenium fixes pass without a retry before the retry wrapper is removed in a later change.
Describe alternatives you've considered
Keeping all tests in runtests.py makes the file difficult to navigate and obscures the distinction between browser, local utility, and container checks.
Splitting the Selenium methods into several new classes, adding framework layers, or changing test discovery would increase scope without addressing the immediate organization problem. Those changes are not part of this proposal.
Removing CI retries immediately would expose the failures but would not fix them.
Additional context
Relevant workflow runs:
- https://github.com/openwisp/docker-openwisp/actions/runs/33426841675 passed all 35 tests on the first attempt after the latest #682 changes.
- https://github.com/openwisp/docker-openwisp/actions/runs/33423534846 passed only on attempt five. The restored websocket, topology, and custom-static tests passed in every attempt, while the recurring admin and password-reset failures caused the retries.
- https://github.com/openwisp/docker-openwisp/actions/runs/33259360357 exhausted all five attempts with the same Redis URL parsing assertion.
- https://github.com/openwisp/docker-openwisp/actions/runs/33185375172 exhausted all five attempts with the earlier topology rendering failure and stale-element failures.
This should be implemented in follow-up PRs rather than expanding #682 beyond its restored Selenium coverage.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/runtests.py and the existing make develop-pythontests entry point; inspect the TestServices, TestLocalUtils, and TestOpenVPN classes and their setup, cleanup, and imports. Reorganize the classes into the named test files while preserving the compatibility runner, then run the targeted TestServices command and the full 35-test command to verify names, lifecycle, and paths remain intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100