knative / knative/eventing

eventing-webhook mutates itself when SinkBinding is present

Open
#8,161 7 comments 0 reactions 0 assignees View on GitHub
kind/bug triage/accepted
Dominant language
Go
Stars
1.6k
Forks
631
Avg merge
2d 1h
Merged PRs (30d)
6

Description

**Describe the bug**
Running downgrade from Eventing 1.16 to 1.15 can fail with "no endpoints available for service "eventing-webhook" when ContainerSource (and thus SinkBinding) is present in the cluster.

The upgrade tests [scale the eventing-webhook](https://github.com/knative/eventing/blob/main/test/e2e-common.sh#L197) to zero and back to 3. But scaling back to 3 fails with this error:
```
executor.go:189: Aug 15 11:44:52.727 install_latest_release [ERR] Error from server (InternalError): Internal error occurred: failed calling webhook "sinkbindings.webhook.sources.knative.dev": failed to call webhook: Post "[https://eventing-webhook.knative-eventing.svc:443/sinkbindings?timeout=10s](https://eventing-webhook.knative-eventing.svc/sinkbindings?timeout=10s)": no endpoints available for service "eventing-webhook"
```
There's a `MutatingWebhookConfiguration` named sinkbindings.webhook.sources.knative.dev which is run by eventing-webhook itself:
```
- apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"admissionregistration.k8s.io/v1","kind":"MutatingWebhookConfiguration","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"knative-eventing","app.kubernetes.io/version":"1.15.0"},"name":"sinkbindings.webhook.sources.knative.dev"},"webhooks":[{"admissionReviewVersions":["v1","v1beta1"],"clientConfig":{"service":{"name":"eventing-webhook","namespace":"knative-eventing"}},"failurePolicy":"Fail","name":"sinkbindings.webhook.sources.knative.dev","sideEffects":"None","timeoutSeconds":10}]}
creationTimestamp: "2024-08-15T07:51:05Z"
generation: 3
labels:
app.kubernetes.io/name: knative-eventing
app.kubernetes.io/version: 1.15.0
name: sinkbindings.webhook.sources.knative.dev
resourceVersion: "13981"
uid: 0a8e964a-e0d9-457c-8ea5-21bcb694f23d
webhooks:
- admissionReviewVersions:
- v1
- v1beta1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNqakNDQWpTZ0F3SUJBZ0lSQUl6V2xXRE8zcHdEa01SZkMwOElTRnN3Q2dZSUtvWkl6ajBFQXdJd1JqRVUKTUJJR0ExVUVDaE1MYTI1aGRHbDJaUzVrWlhZeExqQXNCZ05WQkFNVEpXVjJaVzUwYVc1bkxYZGxZbWh2YjJzdQphMjVoZEdsMlpTMWxkbVZ1ZEdsdVp5NXpkbU13SGhjTk1qUXdPREUxTURjMU1UQTJXaGNOTWpRd09ESXlNRGMxCk1UQTJXakJHTVJRd0VnWURWUVFLRXd0cmJtRjBhWFpsTG1SbGRqRXVNQ3dHQTFVRUF4TWxaWFpsYm5ScGJtY3QKZDJWaWFHOXZheTVyYm1GMGFYWmxMV1YyWlc1MGFXNW5Mbk4yWXpCWk1CTUdCeXFHU000OUFnRUdDQ3FHU000OQpBd0VIQTBJQUJKelc5alhWaFNLRG43Zy9LL3ZmK2traHJhMUhVVTlQRFpyVUVQWDI5YXFmLzNPc09SMCszSTFqCmJEUEJsMmw1OFBsY1BERExSRUEwaFpxc09nSVhOdCtqZ2dFQk1JSCtNQTRHQTFVZER3RUIvd1FFQXdJQ2hEQWQKQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZApCZ05WSFE0RUZnUVUzVTYxODR1MEp4ZU9MUXVLek5FZDY3dzl1dnN3Z1p3R0ExVWRFUVNCbERDQmtZSVFaWFpsCmJuUnBibWN0ZDJWaWFHOXZhNEloWlhabGJuUnBibWN0ZDJWaWFHOXZheTVyYm1GMGFYWmxMV1YyWlc1MGFXNW4KZ2lWbGRtVnVkR2x1WnkxM1pXSm9iMjlyTG10dVlYUnBkbVV0WlhabGJuUnBibWN1YzNaamdqTmxkbVZ1ZEdsdQpaeTEzWldKb2IyOXJMbXR1WVhScGRtVXRaWFpsYm5ScGJtY3VjM1pqTG1Oc2RYTjBaWEl1Ykc5allXd3dDZ1lJCktvWkl6ajBFQXdJRFNBQXdSUUlnRXNEWk1sY3NXTzZZcDlrRWoyZTZRYzRRZTBlT1RwWmRmeXF6TkpLMnZXSUMKSVFDYUc2azNHYk1FZnhIakFqeU5IWTdyME5GNFRMSG1KWG5jdzZOVURrUjc1QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
service:
name: eventing-webhook
namespace: knative-eventing
path: /sinkbindings
port: 443
failurePolicy: Fail
matchPolicy: Equivalent
name: sinkbindings.webhook.sources.knative.dev
namespaceSelector:
matchExpressions:
- key: bindings.knative.dev/exclude
operator: NotIn
values:
- "true"
objectSelector:
matchExpressions:
- key: bindings.knative.dev/exclude
operator: NotIn
values:
- "true"
reinvocationPolicy: IfNeeded
rules:
- apiGroups:
- apps
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- deployments/*
scope: '*'
sideEffects: None
timeoutSeconds: 10
```
As a result, the eventing-webhook is not started again and remains scaled to zero.

It happens on [this PR](https://github.com/knative/eventing/pull/8158) which extends upgrade/downgrade tests in a specific way. Some resources are created before upgrade and verified after upgrade, and some resources are created after upgrade and verified later after downgrade. The tests now include ContainerSource which creates SinkBindings.

Example failure is in [this run](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_eventing/8158/upgrade-tests_eventing_main/1823987656074203136)

However, the eventing-webhook Deployment already has the label `bindings.knative.dev/exclude: "true"` which should exclude it from the webhook selection:
```
- apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"eventing-webhook","app.kubernetes.io/name":"knative-eventing","app.kubernetes.io/version":"1.15.0","bindings.knative.dev/exclude":"true"},"name":"eventing-webhook","namespace":"knative-eventing"},"spec":{"selector":{"matchLabels":{"app":"eventing-webhook","role":"eventing-webhook"}},"template":{"metadata":{"labels":{"app":"eventing-webhook","app.kubernetes.io/component":"eventing-webhook","app.kubernetes.io/name":"knative-eventing","app.kubernetes.io/version":"1.15.0","role":"eventing-webhook"}},"spec":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}],"image":"gcr.io/knative-releases/knative.dev/eventing/cmd/webhook@sha256:086fc8a0da40730bbfcc32cb12ebc5c883fe75ff05be1bba54435ef1fdc080ce","livenessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}],"port":8443,"scheme":"HTTPS"},"initialDelaySeconds":120,"periodSeconds":1},"name":"eventing-webhook","ports":[{"containerPort":8443,"name":"https-webhook"},{"containerPort":9090,"name":"metrics"},{"containerPort":8008,"name":"profiling"}],"readinessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}],"port":8443,"scheme":"HTTPS"},"periodSeconds":1},"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"100m","memory":"50Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true,"runAsNonRoot":true,"seccompProfile":{"type":"RuntimeDefault"}},"terminationMessagePolicy":"FallbackToLogsOnError"}],"enableServiceLinks":false,"serviceAccountName":"eventing-webhook","terminationGracePeriodSeconds":300}}}}
creationTimestamp: "2024-08-15T07:50:56Z"
generation: 10
labels:
app.kubernetes.io/component: eventing-webhook
app.kubernetes.io/name: knative-eventing
app.kubernetes.io/version: 1.15.0
bindings.knative.dev/exclude: "true"
name: eventing-webhook
namespace: knative-eventing
resourceVersion: "14396"
uid: 397f7f2f-8753-4de6-892b-eb63c42e9059
spec:
progressDeadlineSeconds: 600
replicas: 0
```

**Expected behavior**
The eventing-webhook is up-and-running after downgrade.

**To Reproduce**
Having ContainerSource and doing downgrade from 1.16 to 1.15. [This PR](https://github.com/knative/eventing/pull/8158) reproduces the behavior reliably.

**Knative release version**
Downgrading from pre-release 1.16 to 1.15

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the upgrade test setup in test/e2e-common.sh around line 197 and the upgrade/downgrade scenario described in PR 8158. Inspect the eventing-webhook Deployment labels and the sinkbindings MutatingWebhookConfiguration when ContainerSource creates SinkBindings. Done means the webhook comes back up after downgrading from 1.16 to 1.15 and the upgrade tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.