Publish workflow should validate OCI image annotations after build

What happened

On PR #3124, the /publish command at 17:08 UTC built and pushed 2 OCI images. The publish workflow reported success and listed both images in its PR comment. However, the backend image (backstage-community-plugin-servicenow-backend:pr_3124__1.13.1) was missing the required io.backstage.dynamic-packages annotation. This was only discovered when smoke tests ran at 17:16 and failed with InstallException: No plugins found in OCI image. The same smoke test failure recurred 3 more times (run 30932951763, run 30938766421, run 30939066031) before the human merged main and re-published at 18:37, which produced a valid image. Total wasted time: ~90 minutes.

What could go better

The publish workflow currently validates metadata files (the PR comment says 'Metadata Validation: All metadata files validated successfully') but does not inspect the annotations on the OCI images it produces. The io.backstage.dynamic-packages annotation is a hard requirement for RHDH to load the plugin — without it, the image is unusable. The publish workflow should verify this annotation exists on each image immediately after pushing, before reporting success. This would have caught the malformed backend image at 17:13 instead of letting it propagate to 4 failed smoke test runs.

Confidence: high. The annotation check is a deterministic post-push validation — skopeo inspect or oras manifest fetch can read OCI annotations without pulling the full image. The existing publish workflow already has the image references available since it lists them in its success comment. This is distinct from issue #3061 (which covers plugins-list entries that produce no artifact) and #3122 (which covers pre-build metadata validation) — this proposal is specifically about post-build annotation validation on produced artifacts.

Proposed change

In the publish workflow (publish-workspace-plugins.yaml or the underlying export script from rhdh-plugin-export-utils), add a post-push validation step after all OCI images are built and pushed. For each published image:

1. Run skopeo inspect --raw docker://<image-ref> (or equivalent) to read the OCI manifest annotations.
2. Verify the io.backstage.dynamic-packages annotation is present and non-empty.
3. If any image fails validation, mark the publish as failed (not just a warning) and include the specific image reference and missing annotation in the PR comment.

The validation should run before the success comment is posted. A failed annotation check should produce a clear error like: Error: Image backstage-community-plugin-servicenow-backend:pr_3124__1.13.1 was pushed but is missing the required 'io.backstage.dynamic-packages' annotation. The image cannot be loaded by RHDH. This may indicate a CLI version mismatch — try merging the base branch and re-running /publish.

Validation criteria

After implementation, intentionally publish an image with a known annotation issue (e.g., by using a misconfigured CLI version or --embed-package on a plugin that doesn't support it). The publish workflow should fail and report the missing annotation in the PR comment instead of reporting success. Verify on the next 3 workspace update PRs that use /publish: all should either pass the annotation check cleanly or fail fast with a clear error. No smoke test run should encounter the io.backstage.dynamic-packages annotation error on an image that the publish workflow reported as successfully built.

Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3124

Start by reading publish-workspace-plugins.yaml and tracing the underlying export script from rhdh-plugin-export-utils to see where published image references and the success comment are assembled. Run skopeo inspect --raw against a published image and identify how to check its annotations before the success comment. Done means each image is checked for a non-empty io.backstage.dynamic-packages annotation, and a failure names the image and appears in the PR comment.