canonical / canonical/github-runner-image-builder-operator

Partial multi-cloud upload failure discards successful uploads and skips relation-data update, risking stale/pruned images

Open
#241 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2
Forks
3
Avg merge
21h 36m
Merged PRs (30d)
2

Description

### Bug Description

When `--upload-clouds` is configured with more than one cloud, `_upload_to_clouds()` in `app/src/github_runner_image_builder/openstack_builder.py` (lines ~840-877) uploads to each cloud sequentially in a loop, calling `store.upload_image()` per cloud. If a single cloud's upload raises `UploadImageError` (e.g. a network blip), the exception propagates immediately and the function never returns — discarding the `images` list entries for any clouds that already uploaded successfully in that same call.

This exception propagates up through `openstack_builder.run()` → the charm's `builder.run()` (re-raised as `BuilderRunError`) → `charm.py::_run()`:

```python
cloud_images = builder.run(config_matrix=..., static_config=...)
self.image_observer.update_image_data(cloud_images=cloud_images)
```

Because `builder.run()` raises, `update_image_data()` is never called — relation data is **not** updated for *any* cloud, including ones whose upload actually succeeded in this run.

Compounding this, `store.py::upload_image()` (lines 59-92) prunes older images for a cloud immediately after that cloud's own upload succeeds, with no coordination with whether the batch as a whole (or the relation-data update) ever completes:

```python
image = connection.create_image(...)
logger.info("Pruning older images %s, keeping %s.", image_name, keep_revisions)
_prune_old_images(connection=connection, image_name=image_name, num_revisions=keep_revisions)
```

Net effect: if one upload-cloud consistently fails (e.g. a flaky network path to that cloud's Glance endpoint) while another cloud consistently succeeds, every retry keeps uploading + pruning on the healthy cloud but never advances relation data (since the batch always errors out on the unhealthy cloud). After enough successful-but-unrecorded builds on the healthy cloud, `keep_revisions` pruning can delete the very image that relation data still (silently, stale) points to — leaving consumers with a reference to an image that no longer exists in Glance.

Note: `src/image.py::update_image_data()` already supports partial data cleanly — it keys strictly by `cloud_id` and updates each relation independently, so no relation-data model changes are required. The fix is to have the upload path surface partial successes instead of swallowing them via an unhandled exception, and to update relation data with whatever succeeded even when the overall batch partially fails.

### Impact

High (major functionality broken, no workaround)

### Impact Rationale

This caused a full outage of s390x runner spawning on `ps6` (production self-hosted runners) for ~4 days (31 Aug - 04 Sep). The image-builder unit kept retrying every cron cycle, but every run failed while uploading to the `ps7` cross-site cloud (persistent `SSLEOFError` on `glance.ps7.canonical.com`), so relation data was never refreshed even though local `ps6` uploads sometimes succeeded. Runner-manager eventually referenced a pruned/missing image ID, causing `openstack.exceptions.NotFoundException: No Image found for ` and 100% runner spawn failures for the s390x fleet on `ps6`. Root-caused and worked around manually via `relation-set`; see canonical Jira ISD-6561 / ISD-5444 for the incident writeup.

### To Reproduce

1. Deploy `github-runner-image-builder` with `--upload-clouds` pointing at two (or more) OpenStack clouds.
2. Cause uploads to one of the clouds to reliably fail (e.g. block/reset the TLS connection to that cloud's Glance endpoint) while the other cloud remains healthy.
3. Trigger repeated builds (cron `run` event) over several cycles, with `keep_revisions` set low (e.g. 5).
4. Observe: the healthy cloud accumulates new images and prunes old ones each cycle, but the `image` relation data is never updated (every run raises `BuilderRunError` due to the unhealthy cloud), until the once-referenced image on the healthy cloud is eventually pruned away.

### Environment

- Charm: `github-runner-image-builder-operator`, deployed via Juju on OpenStack (multi-cloud upload configuration, s390x arch).
- Affected component: `app/src/github_runner_image_builder/openstack_builder.py::_upload_to_clouds`, `app/src/github_runner_image_builder/store.py::upload_image`, `src/charm.py::_run`.

### Relevant log output

```
2026-09-01 07:47:17,338 - ERROR - Error while uploading image.
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2437)
...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='glance.ps7.canonical.com', port=9292): Max retries exceeded with url: /v2/images/6cc7e346-2a07-46e2-a9ad-6ee109870f19/file (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol')))
...
openstack.exceptions.SDKException: Image creation failed: SSL exception connecting to https://glance.ps7.canonical.com:9292/v2/images/6cc7e346-2a07-46e2-a9ad-6ee109870f19/file: ...

2026-09-04 05:18:00,416 - ERROR - github_runner_manager.manager.runner_manager - Failed to spawn a runner.
...
openstack.exceptions.NotFoundException: No Image found for 944c3715-6c31-486e-a20b-d4bafaf7c18f
```

### Additional context

Suggested fix direction:
- `_upload_to_clouds()`: catch per-cloud `UploadImageError`, continue attempting remaining clouds, and return whatever succeeded (plus signal the partial failure) instead of aborting on the first exception.
- `charm.py::_run()`: call `self.image_observer.update_image_data(cloud_images=cloud_images)` with the partial results before surfacing/logging the overall failure, so relation data always advances for clouds that did succeed.

_AI-generated message._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.