canonical / canonical/checkbox
Checkbox tests get stuck in a loop on `havana-ptz`
- Dominant language
- Python
- Stars
- 52
- Forks
- 81
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 70
Description
### Bug Description
While reviewing `snapd` 2.27, the tests on `havana-ptz` would [get stuck in an infinite loop](https://test-observer.canonical.com/#/snaps/239942?Review+status=Undecided&Environment=havana-ptz), repeating the same tests over and over every minute or so:
Installing Checkbox from `stable` instead of `edge` resolved the issue. Here are the current versions for `checkbox20`.
- latest/stable: 6.0.0-dev90 2025-09-22 (2828) 278MB
- latest/edge: 7.0.0-dev19 2025-10-06 (2873) 278MB
### Cert-blocker Test Case
- [ ] cert-blocker
### To Reproduce
I haven't tried to isolate/reproduce the issue but:
- This is a [run](http://10.102.156.15:8080/job/cert-havana-ptz-snapd-beta/79/) where the tests are stuck in a loop, with `checkbox-havana` (and subsequently the `checkbox20` runtime) installed from `edge`
- This is a [run](http://10.102.156.15:8080/job/cert-havana-ptz-snapd-beta/80/) where the tests are _not_ stuck in a loop, with `checkbox-havana` (and subsequently the `checkbox20` runtime) installed from `stable`
### Expected Result
The job completes.
### Actual Result
The job times out because it's stuck in a specific 21-test loop.
### Environment
- UC20
- Checkbox installed from a snap: `checkbox-havana` -> `checkbox20`
- latest/stable: 6.0.0-dev90 2025-09-22 (2828) 278MB
- latest/edge: 7.0.0-dev19 2025-10-06 (2873) 278MB
- Environment: `havana-ptz`
- Under test: `snapd` 2.72
### Relevant log output
```shell
# Over and over again ad infinitum
13:28:29 -----------------------------[ Running job 1 / 21 ]-----------------------------
13:28:29 -----[ Refresh hbt-sec-camera-s70-icptz-gadget snap to its base revision ]------
13:28:29 ID: com.canonical.certification::snapd/snap-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 2 / 21 ]-----------------------------
13:28:29 [ Verify hbt-sec-camera-s70-icptz-gadget snap revision after refreshing to base revision ]
13:28:29 ID: com.canonical.certification::snapd/snap-verify-after-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 3 / 21 ]-----------------------------
13:28:29 [ Refresh hbt-sec-camera-s70-icptz-gadget snap to latest revision in stable channel ]
13:28:29 ID: com.canonical.certification::snapd/snap-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-stable-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 4 / 21 ]-----------------------------
13:28:29 [ Verify hbt-sec-camera-s70-icptz-gadget snap revision after refreshing to latest revision in stable channel ]
13:28:29 ID: com.canonical.certification::snapd/snap-verify-after-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-stable-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 5 / 21 ]-----------------------------
13:28:29 ------------------[ Refresh snapd snap to its base revision ]-------------------
13:28:29 ID: com.canonical.certification::snapd/snap-refresh-snapd-snapd-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 6 / 21 ]-----------------------------
13:28:29 --------[ Verify snapd snap revision after refreshing to base revision ]--------
13:28:29 ID: com.canonical.certification::snapd/snap-verify-after-refresh-snapd-snapd-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 -----------------------------[ Running job 7 / 21 ]-----------------------------
13:29:05 ----------[ Refresh snapd snap to latest revision in stable channel ]-----------
13:29:05 ID: com.canonical.certification::snapd/snap-refresh-snapd-snapd-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 -----------------------------[ Running job 8 / 21 ]-----------------------------
13:29:05 [ Verify snapd snap revision after refreshing to latest revision in stable channel ]
13:29:05 ID: com.canonical.certification::snapd/snap-verify-after-refresh-snapd-snapd-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 -----------------------------[ Running job 9 / 21 ]-----------------------------
13:29:05 -----------[ Refresh hon-hbt-cv22-kernel snap to its base revision ]------------
13:29:05 ID: com.canonical.certification::snapd/snap-refresh-kernel-hon-hbt-cv22-kernel-to-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 10 / 21 ]-----------------------------
13:29:05 -[ Verify hon-hbt-cv22-kernel snap revision after refreshing to base revision ]-
13:29:05 ID: com.canonical.certification::snapd/snap-verify-after-refresh-kernel-hon-hbt-cv22-kernel-to-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 11 / 21 ]-----------------------------
13:29:05 ---[ Refresh hon-hbt-cv22-kernel snap to latest revision in stable channel ]----
13:29:05 ID: com.canonical.certification::snapd/snap-refresh-kernel-hon-hbt-cv22-kernel-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 12 / 21 ]-----------------------------
13:29:05 [ Verify hon-hbt-cv22-kernel snap revision after refreshing to latest revision in stable channel ]
13:29:05 ID: com.canonical.certification::snapd/snap-verify-after-refresh-kernel-hon-hbt-cv22-kernel-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 13 / 21 ]-----------------------------
13:29:05 ----------------------[ clevis encrypt/decrypt precheck ]-----------------------
13:29:05 ID: com.canonical.certification::clevis-encrypt-tpm2/precheck
13:29:05 Category: TPM 2.0 (Trusted Platform Module)
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 14 / 21 ]-----------------------------
13:29:05 [ Revert hbt-sec-camera-s70-icptz-gadget snap from base revision to original revision ]
13:29:05 ID: com.canonical.certification::snapd/snap-revert-gadget-hbt-sec-camera-s70-icptz-gadget-from-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 15 / 21 ]-----------------------------
13:29:05 [ Revert hbt-sec-camera-s70-icptz-gadget snap to original revision from stable channel ]
13:29:05 ID: com.canonical.certification::snapd/snap-revert-gadget-hbt-sec-camera-s70-icptz-gadget-from-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 16 / 21 ]-----------------------------
13:29:05 ---------[ Revert snapd snap from base revision to original revision ]----------
13:29:05 ID: com.canonical.certification::snapd/snap-revert-snapd-snapd-from-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 17 / 21 ]-----------------------------
13:29:05 ---------[ Revert snapd snap to original revision from stable channel ]---------
13:29:05 ID: com.canonical.certification::snapd/snap-revert-snapd-snapd-from-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 18 / 21 ]-----------------------------
13:29:05 --[ Revert hon-hbt-cv22-kernel snap from base revision to original revision ]---
13:29:05 ID: com.canonical.certification::snapd/snap-revert-kernel-hon-hbt-cv22-kernel-from-base-rev
13:29:05 Category: Snapd
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
13:29:18 Outcome: job cannot be started
13:29:18 ----------------------------[ Running job 19 / 21 ]-----------------------------
13:29:18 --[ Revert hon-hbt-cv22-kernel snap to original revision from stable channel ]--
13:29:18 ID: com.canonical.certification::snapd/snap-revert-kernel-hon-hbt-cv22-kernel-from-stable-rev
13:29:18 Category: Snapd
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
13:29:18 Outcome: job cannot be started
13:29:18 ----------------------------[ Running job 20 / 21 ]-----------------------------
13:29:18 --------[ Ensure the TPM has required capabilities for clevis RSA test ]--------
13:29:18 ID: com.canonical.certification::clevis-encrypt-tpm2/detect-rsa-capabilities
13:29:18 Category: TPM 2.0 (Trusted Platform Module)
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
13:29:18 Outcome: job cannot be started
13:29:18 ----------------------------[ Running job 21 / 21 ]-----------------------------
13:29:18 --------[ Ensure the TPM has required capabilities for clevis ECC test ]--------
13:29:18 ID: com.canonical.certification::clevis-encrypt-tpm2/detect-ecc-capabilities
13:29:18 Category: TPM 2.0 (Trusted Platform Module)
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
```
Contributor guide
Research direction
Start with the two linked certification runs, comparing the edge installation in run 79 with the stable installation in run 80 and reviewing the repeated 21-test log. Trace the Checkbox job lifecycle around the listed snap refresh, verification, and revert jobs; done means the havana-ptz job completes without repeating the loop or timing out.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100