canonical / canonical/charm-integration-testing
[Enhancement] Stricter historical matching in `test_deploy_target_old_revision`
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
Filed by @canonical/solutions-qa
I was investigating this failure of the `test_deploy_target_old_revision` test:
https://test-observer.canonical.com/#/charms/405107?testExecutionId=475111&testResultId=10557332
The target under test is `postgresql-k8s`. The output shows
```
INFO Found historical revision 495 with passing test_deploy (execution 299462)
```
Test Execution 299462 does indeed have a successful `test_deploy` of `postgresql-k8s` revision 495:
https://test-observer.canonical.com/#/charms/91437?testExecutionId=299462&testResultId=8451864
However, that test result comes from a different test plan. A more direct comparison would come from Test Execution 299509, since that was run for the same exact test plan:
https://test-observer.canonical.com/#/charms/91437?testExecutionId=299509&testResultId=8452225
I think it would be more helpful to select Test Executions from the same test plan where possible, and then potentially fall back to other test plans if there is no match
Contributor guide
Research direction
Start by locating the implementation and tests for `test_deploy_target_old_revision`, then inspect how historical Test Executions are selected and how test plans are represented. Update the behavior so matching executions from the same test plan are preferred, with other plans used as a fallback, and add or adjust coverage showing that preference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100