canonical / canonical/github-runner-image-builder-operator

Integration tests are very flaky

Open
#186 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
3
Avg merge
21h 36m
Merged PRs (30d)
2

Description

### Bug Description

I have noticed that the integration tests `test_charm.py` and `test_upgrade.py` fail quite often as the github-image-builder subprocess command fails to run. For example, in this CI [run](https://github.com/canonical/github-runner-image-builder-operator/actions/runs/21169655156/job/60883390634?pr=185#step:25:401), exceptions.BuilderRunError is raised.

### To Reproduce

Rerun the CI multiple times.

### Environment

-

### Relevant log output

```shell
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/src/builder.py", line 527, in _run
stdout = subprocess.check_output( # pylint: disable=unexpected-keyword-arg # nosec:B603
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/run-one', '/usr/bin/sudo', '--preserve-env', '/home/ubuntu/.local/bin/github-runner-image-builder', '--os-cloud', 'builder', 'run', 'image-builder-operator-531c043c-noble-x64', '--flavor', 'shared.xsmall', '--keep-revisions', '5', '--network', 'net_stg-ps7-github-runner-test-amd64', '--prefix', 'image-builder-operator-531c043c', '--upload-clouds', 'stg-ps7-github-runner-test-amd64_project_stg-ps7-github-runner-test-amd64', '--arch', 'x64', '--base-image', 'noble', '--script-url', 'https://raw.githubusercontent.com/canonical/github-runner-image-builder/refs/heads/main/tests/integration/testdata/test_script.sh', '--proxy', 'egress.ps7.internal:3128']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 338, in wrapped_f
return copy(f, *args, **kw)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 477, in __call__
do = self.iter(retry_state=retry_state)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 378, in iter
result = action(retry_state)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 420, in exc_check
raise retry_exc.reraise()
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 187, in reraise
raise self.last_attempt.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 480, in __call__
result = fn(*args, **kwargs)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/src/builder.py", line 558, in _run
raise BuilderRunError from exc
exceptions.BuilderRunError

Its always this error all the time.
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.