openshift / openshift/machine-config-operator

containers-storage policy key uses the unresolved graph root, so rebase fails when the store is a symlink

Open
#6,388 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
269
Forks
529
Avg merge
3d 18h
Merged PRs (30d)
83

Description

Description

A node degrades during an OS update with is rejected by policy. This needs two things to be true: the container storage graph root is a symlink, and policy.json has a restrictive default.

patchPoliciesForContainerStorage is meant to cover this. It does run. It logs Temporal allow policy added for URL on every attempt and never logs Error writing temporal policy files. The rule it writes just can't be found, because the writer and the evaluator build the scope from different strings:

written   (rpm-ostree.go, from `podman system info`):  [overlay@/var/lib/containers/storage]<repo-digest>@<image-id>
looked up (PolicyConfigurationIdentity):               [overlay@/var/lib/kubelet/containers/storage]<repo-digest>@<image-id>

podman system info gives the graphroot as written in storage.conf. containers/storage runs it through filepath.EvalSymlinks when the store opens, so storageReference.PolicyConfigurationIdentity() and everything in PolicyConfigurationNamespaces() carry the resolved path instead. Nothing matches, so evaluation falls through to default.

Anyone who puts the container store on a separate disk behind a symlink will hit this.

Steps to reproduce the issue:

  1. Relocate the container store: ln -sfT /var/lib/kubelet/containers/storage /var/lib/containers/storage
  2. Set registrySources.allowedRegistries on image.config.openshift.io/cluster, which makes the container runtime config controller render a policy.json with default: reject and no containers-storage transport
  3. With the PinnedImages feature gate on and a PinnedImageSet referenced by the pool, trigger an OS update so MCD takes the local container storage rebase path

Describe the results you received:

rpm-ostree.go] Executing local container storage rebase to <registry>/<repo>@sha256:<digest>
update.go] Running: rpm-ostree rebase --experimental ostree-unverified-image:containers-storage:<registry>/<repo>@sha256:<digest>
update.go] Rolling back applied changes to OS due to error: failed to update OS from local storage: ... error: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: Running image containers-storage:[overlay@/var/lib/kubelet/containers/storage+/run/containers/storage:overlay.skip_mount_home=true]<registry>/<repo>@sha256:<digest>@<image-id> is rejected by policy.
writer.go] Marking Degraded due to: "failed to update OS from local storage: ... is rejected by policy"

The pool sits at 0 updated and 1 degraded, retrying roughly every 80s, and never gets anywhere.

Describe the results you expected:

The rebase works, because the allow rule written for that image is the one the evaluator looks up.

Additional information you deem important:

  • Happens every time, not intermittent.
  • Hard to spot from the logs, since the workaround claims success. grep 'Temporal allow policy added for URL' and look at the graph root in the key.
  • The error return from patchPoliciesForContainerStorage is swallowed on purpose, so a genuine write failure would be silent too.
  • Seen on release-4.20. Same code on main.
  • Fix and regression tests in the linked PR.

Output of oc adm release info --commits | grep machine-config-operator:

4.20.3

Additional environment details (platform, options, etc.):

Bare metal / platform: none, multi-node, RHCOS 9.6, container store relocated to a separate disk via symlink.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rpm-ostree.go at patchPoliciesForContainerStorage, then compare its policy key with storageReference.PolicyConfigurationIdentity() and PolicyConfigurationNamespaces(). Reproduce with a symlinked graph root and restrictive policy.json; done means the temporal allow rule is found by evaluation and the linked regression tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.