canonical / canonical/charm-integration-testing

[Enhancement] Stricter historical matching in `test_deploy_target_old_revision`

Open
#582 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 20h
Merged PRs (30d)
98

Description

Filed by @canonical/solutions-qa

I was investigating this failure of the `test_deploy_target_old_revision` test:
https://test-observer.canonical.com/#/charms/405107?testExecutionId=475111&testResultId=10557332

The target under test is `postgresql-k8s`. The output shows

```
INFO Found historical revision 495 with passing test_deploy (execution 299462)
```

Test Execution 299462 does indeed have a successful `test_deploy` of `postgresql-k8s` revision 495:
https://test-observer.canonical.com/#/charms/91437?testExecutionId=299462&testResultId=8451864

However, that test result comes from a different test plan. A more direct comparison would come from Test Execution 299509, since that was run for the same exact test plan:
https://test-observer.canonical.com/#/charms/91437?testExecutionId=299509&testResultId=8452225

I think it would be more helpful to select Test Executions from the same test plan where possible, and then potentially fall back to other test plans if there is no match

Contributor guide

Open the contributing guide

Research direction

Start by locating the implementation and tests for `test_deploy_target_old_revision`, then inspect how historical Test Executions are selected and how test plans are represented. Update the behavior so matching executions from the same test plan are preferred, with other plans used as a fallback, and add or adjust coverage showing that preference.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.