openshift / openshift/machine-config-operator
containers-storage policy key uses the unresolved graph root, so rebase fails when the store is a symlink
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 269
- Forks
- 529
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 83
Description
Description
A node degrades during an OS update with is rejected by policy. This needs two things to be true: the container storage graph root is a symlink, and policy.json has a restrictive default.
patchPoliciesForContainerStorage is meant to cover this. It does run. It logs Temporal allow policy added for URL on every attempt and never logs Error writing temporal policy files. The rule it writes just can't be found, because the writer and the evaluator build the scope from different strings:
written (rpm-ostree.go, from `podman system info`): [overlay@/var/lib/containers/storage]<repo-digest>@<image-id>
looked up (PolicyConfigurationIdentity): [overlay@/var/lib/kubelet/containers/storage]<repo-digest>@<image-id>
podman system info gives the graphroot as written in storage.conf. containers/storage runs it through filepath.EvalSymlinks when the store opens, so storageReference.PolicyConfigurationIdentity() and everything in PolicyConfigurationNamespaces() carry the resolved path instead. Nothing matches, so evaluation falls through to default.
Anyone who puts the container store on a separate disk behind a symlink will hit this.
Steps to reproduce the issue:
- Relocate the container store:
ln -sfT /var/lib/kubelet/containers/storage /var/lib/containers/storage - Set
registrySources.allowedRegistriesonimage.config.openshift.io/cluster, which makes the container runtime config controller render apolicy.jsonwithdefault: rejectand nocontainers-storagetransport - With the
PinnedImagesfeature gate on and aPinnedImageSetreferenced by the pool, trigger an OS update so MCD takes the local container storage rebase path
Describe the results you received:
rpm-ostree.go] Executing local container storage rebase to <registry>/<repo>@sha256:<digest>
update.go] Running: rpm-ostree rebase --experimental ostree-unverified-image:containers-storage:<registry>/<repo>@sha256:<digest>
update.go] Rolling back applied changes to OS due to error: failed to update OS from local storage: ... error: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: Running image containers-storage:[overlay@/var/lib/kubelet/containers/storage+/run/containers/storage:overlay.skip_mount_home=true]<registry>/<repo>@sha256:<digest>@<image-id> is rejected by policy.
writer.go] Marking Degraded due to: "failed to update OS from local storage: ... is rejected by policy"
The pool sits at 0 updated and 1 degraded, retrying roughly every 80s, and never gets anywhere.
Describe the results you expected:
The rebase works, because the allow rule written for that image is the one the evaluator looks up.
Additional information you deem important:
- Happens every time, not intermittent.
- Hard to spot from the logs, since the workaround claims success.
grep 'Temporal allow policy added for URL'and look at the graph root in the key. - The error return from
patchPoliciesForContainerStorageis swallowed on purpose, so a genuine write failure would be silent too. - Seen on
release-4.20. Same code onmain. - Fix and regression tests in the linked PR.
Output of oc adm release info --commits | grep machine-config-operator:
4.20.3
Additional environment details (platform, options, etc.):
Bare metal / platform: none, multi-node, RHCOS 9.6, container store relocated to a separate disk via symlink.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rpm-ostree.go at patchPoliciesForContainerStorage, then compare its policy key with storageReference.PolicyConfigurationIdentity() and PolicyConfigurationNamespaces(). Reproduce with a symlinked graph root and restrictive policy.json; done means the temporal allow rule is found by evaluation and the linked regression tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100