canonical / canonical/checkbox

Checkbox tests get stuck in a loop on `havana-ptz`

Open
#2,145 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
52
Forks
81
Avg merge
2d 18h
Merged PRs (30d)
70

Description

### Bug Description

While reviewing `snapd` 2.27, the tests on `havana-ptz` would [get stuck in an infinite loop](https://test-observer.canonical.com/#/snaps/239942?Review+status=Undecided&Environment=havana-ptz), repeating the same tests over and over every minute or so:

Image

Image

Installing Checkbox from `stable` instead of `edge` resolved the issue. Here are the current versions for `checkbox20`.

- latest/stable: 6.0.0-dev90 2025-09-22 (2828) 278MB
- latest/edge: 7.0.0-dev19 2025-10-06 (2873) 278MB

### Cert-blocker Test Case

- [ ] cert-blocker

### To Reproduce

I haven't tried to isolate/reproduce the issue but:

- This is a [run](http://10.102.156.15:8080/job/cert-havana-ptz-snapd-beta/79/) where the tests are stuck in a loop, with `checkbox-havana` (and subsequently the `checkbox20` runtime) installed from `edge`
- This is a [run](http://10.102.156.15:8080/job/cert-havana-ptz-snapd-beta/80/) where the tests are _not_ stuck in a loop, with `checkbox-havana` (and subsequently the `checkbox20` runtime) installed from `stable`

### Expected Result

The job completes.

### Actual Result

The job times out because it's stuck in a specific 21-test loop.

### Environment

- UC20
- Checkbox installed from a snap: `checkbox-havana` -> `checkbox20`
- latest/stable: 6.0.0-dev90 2025-09-22 (2828) 278MB
- latest/edge: 7.0.0-dev19 2025-10-06 (2873) 278MB
- Environment: `havana-ptz`
- Under test: `snapd` 2.72

### Relevant log output

```shell
# Over and over again ad infinitum
13:28:29 -----------------------------[ Running job 1 / 21 ]-----------------------------
13:28:29 -----[ Refresh hbt-sec-camera-s70-icptz-gadget snap to its base revision ]------
13:28:29 ID: com.canonical.certification::snapd/snap-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 2 / 21 ]-----------------------------
13:28:29 [ Verify hbt-sec-camera-s70-icptz-gadget snap revision after refreshing to base revision ]
13:28:29 ID: com.canonical.certification::snapd/snap-verify-after-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 3 / 21 ]-----------------------------
13:28:29 [ Refresh hbt-sec-camera-s70-icptz-gadget snap to latest revision in stable channel ]
13:28:29 ID: com.canonical.certification::snapd/snap-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-stable-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 4 / 21 ]-----------------------------
13:28:29 [ Verify hbt-sec-camera-s70-icptz-gadget snap revision after refreshing to latest revision in stable channel ]
13:28:29 ID: com.canonical.certification::snapd/snap-verify-after-refresh-gadget-hbt-sec-camera-s70-icptz-gadget-to-stable-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 5 / 21 ]-----------------------------
13:28:29 ------------------[ Refresh snapd snap to its base revision ]-------------------
13:28:29 ID: com.canonical.certification::snapd/snap-refresh-snapd-snapd-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:28:29 Outcome: job cannot be started
13:28:29 -----------------------------[ Running job 6 / 21 ]-----------------------------
13:28:29 --------[ Verify snapd snap revision after refreshing to base revision ]--------
13:28:29 ID: com.canonical.certification::snapd/snap-verify-after-refresh-snapd-snapd-to-base-rev
13:28:29 Category: Snapd
13:28:29 --------------------------------------------------------------------------------
13:28:29 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 -----------------------------[ Running job 7 / 21 ]-----------------------------
13:29:05 ----------[ Refresh snapd snap to latest revision in stable channel ]-----------
13:29:05 ID: com.canonical.certification::snapd/snap-refresh-snapd-snapd-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 -----------------------------[ Running job 8 / 21 ]-----------------------------
13:29:05 [ Verify snapd snap revision after refreshing to latest revision in stable channel ]
13:29:05 ID: com.canonical.certification::snapd/snap-verify-after-refresh-snapd-snapd-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 -----------------------------[ Running job 9 / 21 ]-----------------------------
13:29:05 -----------[ Refresh hon-hbt-cv22-kernel snap to its base revision ]------------
13:29:05 ID: com.canonical.certification::snapd/snap-refresh-kernel-hon-hbt-cv22-kernel-to-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 10 / 21 ]-----------------------------
13:29:05 -[ Verify hon-hbt-cv22-kernel snap revision after refreshing to base revision ]-
13:29:05 ID: com.canonical.certification::snapd/snap-verify-after-refresh-kernel-hon-hbt-cv22-kernel-to-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 11 / 21 ]-----------------------------
13:29:05 ---[ Refresh hon-hbt-cv22-kernel snap to latest revision in stable channel ]----
13:29:05 ID: com.canonical.certification::snapd/snap-refresh-kernel-hon-hbt-cv22-kernel-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 12 / 21 ]-----------------------------
13:29:05 [ Verify hon-hbt-cv22-kernel snap revision after refreshing to latest revision in stable channel ]
13:29:05 ID: com.canonical.certification::snapd/snap-verify-after-refresh-kernel-hon-hbt-cv22-kernel-to-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 13 / 21 ]-----------------------------
13:29:05 ----------------------[ clevis encrypt/decrypt precheck ]-----------------------
13:29:05 ID: com.canonical.certification::clevis-encrypt-tpm2/precheck
13:29:05 Category: TPM 2.0 (Trusted Platform Module)
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 14 / 21 ]-----------------------------
13:29:05 [ Revert hbt-sec-camera-s70-icptz-gadget snap from base revision to original revision ]
13:29:05 ID: com.canonical.certification::snapd/snap-revert-gadget-hbt-sec-camera-s70-icptz-gadget-from-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 15 / 21 ]-----------------------------
13:29:05 [ Revert hbt-sec-camera-s70-icptz-gadget snap to original revision from stable channel ]
13:29:05 ID: com.canonical.certification::snapd/snap-revert-gadget-hbt-sec-camera-s70-icptz-gadget-from-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 16 / 21 ]-----------------------------
13:29:05 ---------[ Revert snapd snap from base revision to original revision ]----------
13:29:05 ID: com.canonical.certification::snapd/snap-revert-snapd-snapd-from-base-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 17 / 21 ]-----------------------------
13:29:05 ---------[ Revert snapd snap to original revision from stable channel ]---------
13:29:05 ID: com.canonical.certification::snapd/snap-revert-snapd-snapd-from-stable-rev
13:29:05 Category: Snapd
13:29:05 --------------------------------------------------------------------------------
13:29:05 --------------------------------------------------------------------------------
13:29:05 Outcome: job cannot be started
13:29:05 ----------------------------[ Running job 18 / 21 ]-----------------------------
13:29:05 --[ Revert hon-hbt-cv22-kernel snap from base revision to original revision ]---
13:29:05 ID: com.canonical.certification::snapd/snap-revert-kernel-hon-hbt-cv22-kernel-from-base-rev
13:29:05 Category: Snapd
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
13:29:18 Outcome: job cannot be started
13:29:18 ----------------------------[ Running job 19 / 21 ]-----------------------------
13:29:18 --[ Revert hon-hbt-cv22-kernel snap to original revision from stable channel ]--
13:29:18 ID: com.canonical.certification::snapd/snap-revert-kernel-hon-hbt-cv22-kernel-from-stable-rev
13:29:18 Category: Snapd
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
13:29:18 Outcome: job cannot be started
13:29:18 ----------------------------[ Running job 20 / 21 ]-----------------------------
13:29:18 --------[ Ensure the TPM has required capabilities for clevis RSA test ]--------
13:29:18 ID: com.canonical.certification::clevis-encrypt-tpm2/detect-rsa-capabilities
13:29:18 Category: TPM 2.0 (Trusted Platform Module)
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
13:29:18 Outcome: job cannot be started
13:29:18 ----------------------------[ Running job 21 / 21 ]-----------------------------
13:29:18 --------[ Ensure the TPM has required capabilities for clevis ECC test ]--------
13:29:18 ID: com.canonical.certification::clevis-encrypt-tpm2/detect-ecc-capabilities
13:29:18 Category: TPM 2.0 (Trusted Platform Module)
13:29:18 --------------------------------------------------------------------------------
13:29:18 --------------------------------------------------------------------------------
```

Contributor guide

Open the contributing guide

Research direction

Start with the two linked certification runs, comparing the edge installation in run 79 with the stable installation in run 80 and reviewing the repeated 21-test log. Trace the Checkbox job lifecycle around the listed snap refresh, verification, and revert jobs; done means the havana-ptz job completes without repeating the loop or timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.