ansible / ansible/ansible-dev-tools

feat: Add end-to-end test suite for bundled tools workflow

Open
#765 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
206
Forks
71
Avg merge
4d 6h
Merged PRs (30d)
4

Description

## Summary

Propose adding an end-to-end test suite that validates all bundled tools work together in a **Create → Test → Deploy** workflow. The existing test suite covers unit tests for CLI/server and integration tests for the REST API, but there are no tests that exercise the tools chaining together as a developer would use them.

This proposal is derived from the [LB2236 lab](https://github.com/rhpds/ansible-dev-tools-showroom) solver playbooks, which automate the full developer workflow across all 10 bundled tools. The idea is to translate that workflow coverage into pytest integration tests that follow the existing upstream patterns.

## Motivation

- The bundled tools have individual test suites, but **cross-tool integration is untested** — a breaking change in ansible-creator's output structure could silently break ade, molecule, or tox-ansible downstream.
- The lab solver playbooks already caught real bugs (e.g., `ade install -e .` causing tarball bloat in `ansible-galaxy collection build`, ansible-creator CLI syntax changes breaking scaffolding).
- An e2e suite would serve as a **regression gate** for the meta-package — ensuring the tools continue to work together across releases.

## Tool Coverage

| Tool | What the e2e tests would validate |
|---|---|
| ansible-creator | `init collection`, `init playbook`, `add plugin` |
| ansible-dev-environment (ade) | `install -e .`, `tree`, dependency resolution, venv structure |
| ansible-lint | Violation detection, `--fix` auto-correction, clean pass |
| molecule | Scaffolded scenario execution (`molecule test`) |
| pytest-ansible | `ansible_module` fixture with custom module |
| tox-ansible | Environment discovery (`tox --ansible list`), galaxy validation |
| ansible-galaxy | `collection build`, tarball structure, `install`, `list` |
| ansible-builder | `create` (Containerfile generation), `build` (container-gated) |
| ansible-navigator | `run` in stdout mode, `collections` listing |
| ansible-sign | GPG key generation, `gpg-sign`, `gpg-verify`, tampering detection |

## Design

### Architecture

- **Function-based pytest tests** following the existing upstream patterns (no class-based tests)
- **Session-scoped fixture chain** for the expensive collection lifecycle — scaffold once, share across tests:

```
collection_project (ansible-creator)
└── collection_with_deps (ade install)
└── collection_with_module (cowsay module + lint)
└── collection_tarball (ansible-galaxy build)
```

- **New `@pytest.mark.e2e` marker** with `--include-e2e` / `--only-e2e` CLI options — same pattern as `@pytest.mark.container`
- **`strip_ansi_escape()` + `NO_COLOR=1`** on all CLI output, consistent with existing tests
- **Container-dependent tests** (builder image build) double-marked with `@pytest.mark.container`
- **Self-contained cowsay module** that generates art inline — no external `cowsay` binary dependency

### File structure

```
src/ansible_dev_tools/tests/
├── conftest.py # MODIFY — add e2e marker + CLI options
├── fixtures/integration/e2e/ # CREATE — fixture data files
├── integration/
│ ├── conftest.py # MODIFY — add shared e2e fixtures
│ ├── test_e2e_creator.py # CREATE
│ ├── test_e2e_ade.py # CREATE
│ ├── test_e2e_lint.py # CREATE
│ ├── test_e2e_molecule.py # CREATE
│ ├── test_e2e_pytest_ansible.py # CREATE
│ ├── test_e2e_tox_ansible.py # CREATE
│ ├── test_e2e_galaxy.py # CREATE
│ ├── test_e2e_builder.py # CREATE
│ ├── test_e2e_navigator.py # CREATE
│ └── test_e2e_sign.py # CREATE
```

### tox environment

A new `tox -e e2e` environment that runs `pytest --include-e2e -k e2e`.

### What this does NOT cover

- `ansible-galaxy collection publish` — would need a mock Galaxy server
- ansible-navigator interactive TUI mode — already covered by existing `ContainerTmux` tests
- ansible-compat — library with no CLI, exercised indirectly through all tools
- CI workflow changes — would be a follow-up PR

## Implementation

I have a detailed implementation plan with full test code and can open a PR against this repo. Happy to adjust the approach based on maintainer feedback.

## References

- Lab solver playbooks: [rhpds/ansible-dev-tools-showroom/solvers/](https://github.com/rhpds/ansible-dev-tools-showroom/tree/main/solvers)
- Solver testing report documenting known issues and workarounds found during lab validation

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing pytest integration patterns in src/ansible_dev_tools/tests/integration/ and the root conftest.py, then compare the proposed fixture chain with the LB2236 solver playbooks. Define the shared e2e fixtures, marker and tox environment across the listed files, and verify that the Create → Test → Deploy workflow covers the stated tools and passes with the documented container gating.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.