canonical / canonical/cloud-init
[enhancement]: system-dependent unit tests
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 1.1k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 18
Description
# Enhancement
While many improvements have been made to make unit tests more independent of the runtime environment, there is more work to be done. A low-effort smoke test for unintentional system dependencies is to run them on a non-Linux Unix. GH Actions makes it very easy to test on macos. Using the following diff:
```diff
commit 33dba15d07448c714b57e5eacebd54907a57cea1 (HEAD -> main)
Author: Brett Holman
Date: Mon May 27 15:33:59 2024 -0600
run on macos
diff --git a/.github/workflows/unit.yml b/.github/workflows/unit.yml
index ec054f86c..67472b1f2 100644
--- a/.github/workflows/unit.yml
+++ b/.github/workflows/unit.yml
@@ -14,24 +14,19 @@ jobs:
unittests:
strategy:
matrix:
- python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
+ python-version: ["3.10", "3.11", "3.12"]
toxenv: [py3]
experimental: [false]
check-latest: [false]
continue-on-error: [false]
include:
- - python-version: "3.6"
- toxenv: lowest-supported
- continue-on-error: false
- check-latest: false
- experimental: false
- python-version: "3.13-dev"
toxenv: py3
check-latest: true
experimental: true
continue-on-error: true
name: unittest / ${{ matrix.toxenv }} / python ${{matrix.python-version}}
- runs-on: ubuntu-20.04
+ runs-on: macos-latest
continue-on-error: ${{ matrix.experimental }}
steps:
- name: "Checkout"
```
I see 37 failures:
```
= 37 failed, 5217 passed, 11 skipped, 1 deselected, 10 xfailed, 315 warnings in 62.78s (0:01:02) =
py3: exit 1 (64.80 seconds) /Users/runner/work/cloud-init/cloud-init> .tox/py3/bin/python -m pytest -vvvv --showlocals --durations 10 -m 'not hypothesis_slow' --cov=cloudinit --cov-branch tests/unittests pid=4525
py3: FAIL code 1 (68.78=setup[3.98]+cmd[64.80] seconds)
evaluation failed :( (68.82 seconds)
```
See the full log:
[macos-unit-test-failures.log](https://github.com/canonical/cloud-init/files/15462158/macos-unit-test-failures.log)
Most of these failures are the result of assuming temporary file locations, filepath names, subcommand differences, and a couple are just due to unchangeable things (ex: AF_SOCKET not existing on macos) - all trivial things to fix. While cloud-init has no intention of running on macos, a Unix environment such as macos is a cheap way to get coverage of these tests, and would allow developers on macos to more easily contribute to this project.
Contributor guide
Assessment
This issue has not been assessed yet.