knative / knative/eventing

Removed subscriptions are still being "reconciled", as their references are not being removed from the tracker

Open Beginner friendly
#9,103 1 comment 1 reaction 0 assignees View on GitHub
kind/bug
Dominant language
Go
Stars
1.6k
Forks
631
Avg merge
2d 1h
Merged PRs (30d)
6

Description

**Describe the bug**
Having a Channel in which Subscriptions are added and removed frequently, we see the "Reconcile succeeded" being logged even for removed subscriptions.

**Expected behavior**
removed subscriptions should not be "reconciled" anymore when the Channel is

**To Reproduce**
Adding and removing subscriptions, notice the "Reconcile succeeded" messages in logs for the already deleted Subscriptions

**Knative release version**
1.21

**Additional context**
Asking Claude for analysis:

```
---
Root Cause: Missing tracker.OnDeletedObserver Registration

The subscription reconciler's controller (pkg/reconciler/subscription/controller.go:82) registers:

subscriptionInformer.Informer().AddEventHandler(controller.HandleAll(impl.Enqueue))

This uses HandleAll which calls impl.Enqueue for Add, Update, and Delete events. But it never calls impl.Tracker.OnDeletedObserver when a subscription is deleted.

Why this causes the leak

The tracker (vendor/knative.dev/pkg/tracker/enqueue.go) maintains internal maps (exact and inexact) that map tracked references (e.g., Channels) back to the observers watching them (e.g., Subscriptions). Each time a subscription is reconciled, ReconcileKind calls getChannel →
trackAndFetchChannel, which registers the subscription as a tracker observer via:

- r.channelableTracker.TrackKReference(...) (line 361)
- r.tracker.TrackReference(...) (line 414)

When the subscription is deleted, FinalizeKind removes it from the channel's subscriber list, and the finalizer is removed. However, the tracker entries are never cleaned up. The tracker's OnDeletedObserver method (enqueue.go:289-315) exists precisely for this purpose — it
iterates all tracked references and removes the deleted observer's key. But it's never called.

Impact

The tracker lease is 3 * ResyncPeriod = 3 * 10h = 30 hours by default. During that 30-hour window, every time a tracked Channel changes, tracker.OnChanged enqueues the stale subscription keys. The generated reconciler (reconciler.go:221) handles the "not found" case gracefully
(returns nil), but the work is still done — the key is dequeued, the lister is consulted, and a log line is emitted. This is the source of your linearly increasing reconciliation rate: each delete/re-create cycle leaves behind stale tracker entries that keep triggering
reconciliation attempts for 30 hours.

The lazy expiry in OnChanged (enqueue.go:252-258) only cleans up entries when the tracked object (Channel) changes. If the channel is stable, the stale entries simply accumulate in the tracker map without cleanup.

Other controllers do it correctly

For comparison, several other Knative controllers properly register OnDeletedObserver:

- knative.dev/serving/pkg/reconciler/route/controller.go:110 — DeleteFunc: c.tracker.OnDeletedObserver
- knative.dev/serving/pkg/reconciler/certificate/controller.go:108 — DeleteFunc: c.tracker.OnDeletedObserver
- knative.dev/serving/pkg/reconciler/labeler/controller.go:65 — DeleteFunc: impl.Tracker.OnDeletedObserver
- knative.dev/net-kourier and knative.dev/net-istio do the same

Fix

In pkg/reconciler/subscription/controller.go, line 82 should be changed from:

subscriptionInformer.Informer().AddEventHandler(controller.HandleAll(impl.Enqueue))

to:

subscriptionInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: impl.Enqueue,
UpdateFunc: controller.PassNew(impl.Enqueue),
DeleteFunc: impl.Tracker.OnDeletedObserver,
})

Note: DeleteFunc is set to impl.Tracker.OnDeletedObserver instead of impl.Enqueue. The subscription still gets reconciled on delete because the finalizer mechanism ensures the object goes through FinalizeKind before actual deletion. The DeleteFunc fires only after the object is
removed from the API server (post-finalizer), at which point only tracker cleanup is needed — not another reconciliation.
```

Contributor guide

Open the contributing guide

Research direction

Start in pkg/reconciler/subscription/controller.go around the subscription informer event handler, then read the referenced tracker behavior in vendor/knative.dev/pkg/tracker/enqueue.go. Compare the subscription controller with the other controllers named in the issue and verify that deleted subscriptions no longer produce reconciliation attempts when their Channel changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.