operator-framework / operator-framework/operator-lifecycle-manager

Subscription pointing at a copied CSV becomes permanently orphaned after the original AllNamespaces installation is removed

Open
#3,866 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.9k
Forks
587
Avg merge
2d 8h
Merged PRs (30d)
18

Description

Bug Report

What did you do?

  1. Installed an AllNamespaces operator (sailoperator) in the operators namespace using the pre-existing global-operators OperatorGroup.
  2. OLM copied the CSV into all namespaces.
  3. Created a second namespace (test-sail) with its own OperatorGroup and a Subscription for the same package.
  4. OLM accepted the second Subscription, set status.installedCSV and status.currentCSV, and created an InstallPlan with phase: Complete.
    However, the CSV in test-sail was still the copied CSV (olm.copiedFrom: operators, status.reason: Copied), and no deployment was created in test-sail.
  5. Deleted the original Subscription and CSV from the operators namespace.
  6. OLM garbage-collected all copied CSVs cluster-wide, including the one in test-sail.
# 1. Install AllNamespaces in operators namespace
kubectl apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sailoperator
  namespace: operators
spec:
  channel: stable
  name: sailoperator
  source: operatorhubio-catalog
  sourceNamespace: olm
  installPlanApproval: Automatic
EOF

# 2. Wait for the CSV to install
CSV=$(kubectl get sub sailoperator -n operators -o jsonpath='{.status.currentCSV}' --timeout=60s)
kubectl wait csv "$CSV" -n operators --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s

# 3. Verify OLM copied the CSV into other namespaces
kubectl get csv "$CSV" -n default -o jsonpath='{.metadata.labels.olm\.copiedFrom}'
# operators

# 4. Create second Subscription in a different namespace
kubectl create ns test-sail
kubectl apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-sail-og
  namespace: test-sail
spec:
  targetNamespaces: []
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sailoperator
  namespace: test-sail
spec:
  channel: stable
  name: sailoperator
  source: operatorhubio-catalog
  sourceNamespace: olm
  installPlanApproval: Automatic
EOF

# 5. Verify OLM piggybacked on the copied CSV
kubectl get sub sailoperator -n test-sail -o jsonpath='{.status.installedCSV}'
# (same CSV name as above)
kubectl get csv "$CSV" -n test-sail -o jsonpath='{.metadata.labels.olm\.copiedFrom}'
# operators
kubectl get deployments -n test-sail
# (empty -- no real installation)

# 6. Remove original installation
kubectl delete sub sailoperator -n operators
kubectl delete csv "$CSV" -n operators

# 7. Observe: Subscription is orphaned
kubectl get csv -A | grep sail
# (empty -- all CSVs gone)
kubectl get sub sailoperator -n test-sail -o jsonpath='state={.status.state} installedCSV={.status.installedCSV}'
# state=UpgradePending installedCSV=<stale CSV name>
kubectl get deployments -n test-sail
# (empty)

What did you expect to see?

One of:

  1. OLM reconciles the test-sail Subscription after the copied CSV is garbage-collected and creates a real installation (CSV + deployment) in test-sail.
  2. OLM does not set status.installedCSV on a Subscription that points at a copied CSV it does not own, and instead creates a real installation in test-sail from the start.
  3. OLM reports an error condition on the Subscription indicating the operator is already installed via AllNamespaces in another namespace.

What did you see instead? Under which circumstances?

The test-sail Subscription is permanently orphaned.
It reports installedCSV pointing at a CSV that does not exist anywhere on the cluster.
No error condition is set on the Subscription.
The only condition is CatalogSourcesUnhealthy: AllCatalogSourcesHealthy.

The catalog-operator logs show "skipping update: installplan already created" and "all subscriptions up to date" because the InstallPlan in test-sail is already phase: Complete and approved: true.
The InstallPlan also references the same CSV, which was the copied CSV from the operators namespace, not a real installation in test-sail.

Restarting both olm-operator and catalog-operator does not fix it.
Annotating the Subscription to trigger a reconciliation does not fix it.

Note: this also reproduces without the AllNamespaces scenario.
Installing an operator in a single namespace, then deleting only the CSV, leaves the Subscription in the same orphaned state.
This was reported in #1441 (April 2020) and closed as stale despite a commenter confirming it was still reproducible on OLM 0.16.1.

Environment

  • operator-lifecycle-manager version: 0.43.0 (quay.io/operator-framework/olm@sha256:cbe85a690175f6ae2c140aa8a0e3fbe1a5ffe4cdd2d82191003cd353f71395f3). Changelogs for 0.44.0 and 0.45.0 do not include relevant fixes.
  • Kubernetes version: Server v1.35.0, Client v1.31.1
  • Kubernetes cluster kind: Kind (Kubernetes in Docker)

Possible Solution

The catalog-operator should detect that a Subscription's status.installedCSV references a CSV that no longer exists and trigger a new resolution cycle, regardless of whether the InstallPlan is already complete.

In the AllNamespaces case specifically, OLM should either not set status.installedCSV when the CSV is a copy (olm.copiedFrom label present), or it should detect the loss of the copied CSV and reconcile.

Additional context

The root cause appears to be that once an InstallPlan reaches phase: Complete, the catalog-operator considers the Subscription fully satisfied and does not re-evaluate it, even when the CSV it references is deleted.
In the AllNamespaces variant, this is made worse by OLM pointing the Subscription at a copied CSV that the Subscription has no ownership over, so the CSV can be removed without OLM noticing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the catalog-operator reconciliation path for completed InstallPlans and inspect how it handles a Subscription whose status.installedCSV is missing. Reproduce the AllNamespaces and single-namespace deletion cases from the issue, then add coverage showing that the Subscription recovers or reports an error instead of remaining orphaned.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.