openwisp / openwisp/docker-openwisp

[tests] Stabilize flaky integration tests and reorganize the suite

Open
#683 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
209
Forks
125
Avg merge
1d 4h
Merged PRs (30d)
24

Description

Is your feature request related to a problem? Please describe.
tests/runtests.py contains 1,279 lines across preconditions, dashboard configuration checks, local utility checks, OpenVPN coverage, and 18 functional Selenium tests. Its single TestServices class owns application setup, persistent browser sessions, test fixtures, and unrelated feature checks, which makes it difficult to identify and fix state leaks.

An analysis of the latest 100 CI Build (Docker-OpenWISP) workflow runs from 2026-07-23 through 2026-08-31 found that 89 runs reached the Test step and executed 115 complete Python test-suite attempts. Twenty-eight attempts failed. The CI retry wrapper retried 26 of them, and 10 workflow runs eventually reported success despite one or more failed suite attempts. The retry loop currently reuses the same Compose stack, database, Redis state, volumes, and browser-related test state, so a green workflow does not establish that its first test attempt was clean.

The recurring failures were:

Test Failed results Affected CI runs Failure signature
test_add_superuser 12 8 StaleElementReferenceException
test_forgot_password 10 7 Password-reset confirmation was absent
test_console_errors 6 4 Five stale-element errors and one failed static asset
test_redis_buckets_are_separated 5 1 Deterministic Redis URL parsing assertion
test_topology_graph 3 1 Deterministic topology rendering assertion

The last two rows were deterministic regressions that exhausted all five retries. The Redis parsing issue has since been corrected. The former topology implementation has been replaced by the deterministic database-backed fixture introduced in #682. The first three rows are recurring Selenium synchronization failures that retries conceal.

Describe the solution you'd like
The primary goal is to stabilize the recurring Selenium failures so CI retries no longer conceal them. Reorganize the suite as supporting cleanup: it makes the browser, local utility, and container checks easier to navigate while preserving the existing test lifecycle and targeted-test entry point.

Split the existing test classes into a few direct files while keeping their code and lifecycle intact. Keep tests/runtests.py as a small compatibility runner that imports and re-exports the classes, so make runtests, make develop-pythontests, and python3 tests/runtests.py TestServices.test_name keep working.

tests/
  runtests.py           # Existing entry point and imports
  test_core.py          # Preconditions and dashboard configuration checks
  test_selenium.py      # TestServices unchanged
  test_local_utils.py   # TestLocalUtils unchanged
  test_openvpn.py       # TestOpenVPN unchanged
  utils.py              # Existing shared helpers

The reorganization alone does not isolate state leaks: TestServices will retain its existing class-level setup, persistent browser sessions, and test order. Do not introduce new test framework layers, mixins, fixture abstractions, or a new discovery command. Move each existing class as a whole and retain its current imports, setup, cleanup, and test names.

Address the recurring failures in focused follow-up PRs:

  1. Make user creation and resource selection wait for a fresh clickable element, avoiding the recurring stale-element failures.
  2. Make test_forgot_password wait for its explicit success state instead of inspecting the page source after generic readiness.
  3. Keep the CI retry wrapper until these failures are fixed, but ensure its logs continue to show every failed suite attempt.

Acceptance criteria:

  1. python3 tests/runtests.py TestServices.test_add_superuser continues to resolve and run the named test.
  2. make develop-pythontests continues to execute all 35 tests through tests/runtests.py.
  3. The reorganization preserves class import order, test names, setup, cleanup, and all __file__-relative test paths.
  4. The focused Selenium fixes pass without a retry before the retry wrapper is removed in a later change.

Describe alternatives you've considered
Keeping all tests in runtests.py makes the file difficult to navigate and obscures the distinction between browser, local utility, and container checks.

Splitting the Selenium methods into several new classes, adding framework layers, or changing test discovery would increase scope without addressing the immediate organization problem. Those changes are not part of this proposal.

Removing CI retries immediately would expose the failures but would not fix them.

Additional context
Relevant workflow runs:

This should be implemented in follow-up PRs rather than expanding #682 beyond its restored Selenium coverage.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/runtests.py and the existing make develop-pythontests entry point; inspect the TestServices, TestLocalUtils, and TestOpenVPN classes and their setup, cleanup, and imports. Reorganize the classes into the named test files while preserving the compatibility runner, then run the targeted TestServices command and the full 35-test command to verify names, lifecycle, and paths remain intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, github-actions, python
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.