canonical / canonical/observability-stack

QA infra test improvements

Open
#417 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
9
Forks
12
Avg merge
2d 40m
Merged PRs (30d)
3

Description

For context, first read this HowTo:
- https://github.com/canonical/observability-team/blob/main/how-to/test-products-with-solqa.md

We should improve our QA tests so we have a maintainable implementation and better coverage. This can be done in a few ways:
- Add pytests to our `sqa` folder under [tests](https://github.com/canonical/observability-stack/tree/main/tests), and any other helper functions there. These are executed by the `sqa_tests` in the root repo.
- We should try to avoid having a test variant per product flavor: COS / COS Lite. The main differences are otelcol and timeouts
- We likely don't want to wait 2h for active/idle for COS Lite, although this is valid for COS.
- Consider a good design for checking which flavor we have and piping that into each helper script.
- We should choose which tests we already have in [tests/solution](https://github.com/canonical/observability-stack/tree/main/tests/integration) and port them over to `tests/sqa`, or better yet, make them reusable?
- We should output our results in xUnit format, like how [the Juju team](https://github.com/canonical/terraform-provider-juju-qa/blob/6f49238cfdc1ac3944ff13750f8c4076b2247376/sqa_tests#L52) does it. They do it in GoLang, but we can do it in some Pythonic way.
- This will get rid of [this warning](https://github.com/canonical/terragrunt-deployment-pipelines/actions/runs/28136732680/job/83325143528#step:30:236)
- We could also address [this warning](https://github.com/canonical/terragrunt-deployment-pipelines/actions/runs/28136732680/job/83325143528#step:30:265) by generating a `metrics.json` file (likely a stretch goal).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with how-to/test-products-with-solqa.md, then inspect the existing tests/sqa and tests/integration directories and the root repository's sqa_tests entry point. Compare current COS and COS Lite test variants and identify reusable helpers and coverage to port. Done should include a maintainable test design with appropriate flavor-specific behavior and xUnit output; metrics.json is identified as a stretch goal.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.