cloudposse / cloudposse/atmos

Compatibility-matrix test gate for experimental → stable feature graduation

Open
#2,865 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.4k
Forks
175
Avg merge
2d 3h
Merged PRs (30d)
134

Description

### Describe the Feature

An automated feature-compatibility test matrix that runs as a required CI gate whenever a command's `IsExperimental()` flips to `false` — before a feature loses its `[EXPERIMENTAL]` badge and graduates to stable, CI automatically tests it in combination with a registry of already-stable feature dimensions (vendoring, remote imports, multi-file stacks, custom commands, templating functions, backends, workdir modes, etc.), using pairwise (all-pairs) generation to keep the test count manageable instead of a full combinatorial explosion.

### Expected Behavior

When a PR flips a command's `IsExperimental()` to `false`, CI should auto-generate the pairwise matrix for that feature against the current stable-dimension registry and run the feature's test suite against each generated combination — failing the PR if any combination breaks. Today, nothing like this runs: a feature can go from experimental to stable with no more testing than it already had in isolation.

### Use Case

Isn't it weird to find out that a new stable feature doesn't work with all the old stable features? Say a command ships under `experimental` — it works in a pretty janky, barely-there way for about 1.5 use cases. Fine, that's exactly what experimental is for. But then when it graduates to `stable`, shouldn't there be some kind of check confirming it actually works together with all the already-known, existing features? Right now there isn't.

Atmos' surface area is big and features interact constantly — vendoring + remote imports, workdir + backend provisioning, toolchain + CI, custom commands + stack imports, etc. A feature that looks fine in isolation during its experimental phase (especially if it's only really been stress-tested for ~1-2 use cases by early adopters who click past the `[EXPERIMENTAL]` warning) can still break once combined with everything else people already rely on in stable. That combination is exactly the class of bug that tends to surface only *after* something ships as stable.

The experimental *marking* system itself is genuinely solid — `settings.experimental` (`silence`/`disable`/`warn`/`error`), `ATMOS_EXPERIMENTAL`, `[EXPERIMENTAL]` badges (#1940). But the *graduation* side stops at "we should have one someday" — `docs/prd/experimental-features-system.md` lists a 3-stage path (Experimental → Beta → Stable) explicitly under **Future Considerations** (not implemented), and its own "Testing" section only covers the flag mechanism itself, not cross-feature compatibility. The `tests/` suite isn't thin either (solid CLI/emulator/snapshot tests, e2e workflows like `validation-e2e.yml`/`native-ci.yml`) — it's just organized per-feature, with nothing gating combinations at the graduation moment specifically.

### Describe Ideal Solution

1. **Registry of stable dimensions** — a fixture/config catalog of stable features/configs likely to interact with a graduating feature (vendoring on/off, remote imports, multi-file/parent-scoped stacks, custom commands, templating/YAML functions, multiple backends, workdir modes, toolchain-managed vs system tools, etc.). Raw material already exists: `tests/fixtures`, plus `emulator_cloud_test.go` / `emulator_kubernetes_test.go` / `emulator_native_aws_test.go` / `emulator_openbao_test.go` / `emulator_persistence_test.go` / `emulator_registry_test.go`.
2. **Pairwise (all-pairs) generation** instead of true N×N — covers every *pair* of dimensions with a roughly-linear case count instead of exponential (the standard PICT/allpairs technique for exactly this problem).
3. **Gate tied to `IsExperimental()`** — flipping it to `false` in a PR auto-triggers the matrix; a broken combination blocks the merge.
4. **Phased rollout**: (1) build registry + generator, run informationally against the 5 current experimental commands to baseline; (2) make it blocking specifically for `IsExperimental()` → `false` PRs; (3) keep the registry living as new stable features ship.

### Alternatives Considered

- Documented manual checklist, no automation — too easy to skip/forget, doesn't scale with maintainer time.
- Full N×N combinatorial matrix — correct in theory but explodes fast; pairwise gets most of the bug-catching value for a fraction of the test count.
- Status quo (no formal check, rely on user bug reports post-stable) — works today but graduation quality stays unverified until real users hit it in production.

### Additional Context

- `docs/prd/experimental-features-system.md` — see "Graduation Path" (under Future Considerations) and "Testing"
- #1940 — introduced `settings.experimental` and the `[EXPERIMENTAL]` badge system
- https://atmos.tools/changelog/experimental-feature-flag — current experimental feature list
- Open to LLM-assisted test generation helping bootstrap/maintain the dimension registry and pairwise cases, if useful at this project's scale — happy to discuss either way.

Contributor guide

Open the contributing guide

Research direction

Start with docs/prd/experimental-features-system.md, the tests/fixtures registry, and the named emulator_*_test.go files to understand current experimental coverage and stable dimensions. Done means a pairwise compatibility matrix exists, runs when IsExperimental() changes to false, and blocks the graduation PR when a combination fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.