Sugar Reconciler doesn't delete sugared Brokers when resources or config are updated

**Describe the bug**
The sugar reconciler doesn't delete created Brokers if
- Namespace/Trigger that was responsible for the `default` Broker creation is modified such that it doesn't have the injection label or even deleted
- The `config-sugar.yaml` configmap is updated such that the `namespace-sugar-selector` OR `trigger-sugar-selector` LabelSelectors don't apply to created Brokers.

**Expected behavior**
The Brokers are cleaned up

**To Reproduce**
1. Enable Sugar controller
2. Create a namespace with injection label `eventing.knative.dev/injection: enabled`
3. Wait for a default Broker to be created
4. Modify the namespace and remove the injection label

Also,
5. Same can be be reproduced with using a Trigger for injecting a Broker

**Knative release version**
v1.2

**Additional context**
Add any other context about the problem here such as proposed priority

Start with the Sugar controller and its reconciler, then read config-sugar.yaml and the namespace-sugar-selector and trigger-sugar-selector settings. Reproduce the namespace and Trigger cases described in the issue, and verify that Brokers created by injection are removed when the source resource or selector no longer applies.

Written by the indexing model from the issue text.