Uninstalled Event Source results in "Failed to list *v1.Source: the server could not find the requested resource"

**Describe the bug**
If we create a new Event Source, ie. we install Redis Stream Sources, the parent controller successfully creates a new dynamic controller for this particular Source. But if we uninstall this Source, the created dynamic controller is not properly destroyed. This results in error messages in the log:

```
E1130 20:07:23.537960 1 reflector.go:383] k8s.io/client-go@v11.0.1-0.20190805182717-6502b5e7b1b5+incompatible/tools/cache/reflector.go:125: Failed to watch *v1.Source: the server could not find the requested resource

E1130 20:07:24.782159 1 reflector.go:178] k8s.io/client-go@v11.0.1-0.20190805182717-6502b5e7b1b5+incompatible/tools/cache/reflector.go:125: Failed to list *v1.Source: the server could not find the requested resource

E1130 20:07:27.383944 1 reflector.go:178] k8s.io/client-go@v11.0.1-0.20190805182717-6502b5e7b1b5+incompatible/tools/cache/reflector.go:125: Failed to list *v1.Source: the server could not find the requested resource
```

This code to delete controller is never called: https://github.com/knative/eventing/blob/f4482c580e10dba9711b88e27ccde8b858831756/pkg/reconciler/source/crd/crd.go#L69-L76

**Expected behavior**
Error line below isn't repeated over and over and the appropriate controllers are deleted.

**To Reproduce**
Install knative-eventing
```
k apply --filename https://github.com/knative/eventing/releases/download/v0.19.0/eventing-crds.yaml
k apply --filename https://github.com/knative/eventing/releases/download/v0.19.0/eventing-core.yaml
```

Install any eventing source
For example, RedisStream

```
ko apply -f source/config
```

Delete the eventing source

For example, RedisStream

```
ko delete -f source/config
```

**Knative release version**
0.19.0

**Additional context**
Same problem also in eventing-autoscaler-keda (same code)
https://github.com/knative-sandbox/eventing-autoscaler-keda/issues/2

Start with pkg/reconciler/source/crd/crd.go at lines 69-76 and reproduce the issue using the eventing 0.19.0 and RedisStream source install/delete commands in the report. Trace why controller deletion is not reached, then verify that uninstalling a source removes its controller and stops the repeated list/watch errors.

Written by the indexing model from the issue text.