Inform user of dropped events
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 631
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 6
Description
**Problem**
Whenever events are being dropped by event mesh, we should inform the end-user about that. Just logging is not enough for such a case. IMHO, the best to create a dedicated warning level Kubernetes event, which is designed for such a case: https://www.cncf.io/blog/2023/03/13/how-to-use-kubernetes-events-for-effective-alerting-and-monitoring/
**Background**
With the recent addition of the EventTransform API, the chance of misconfiguration by the end-user raises greatly, as they could easily create infinite loops by transforming the source event in a way the original trigger matches again. The TTL mechanism should eventually break that loop, but in that case, we should inform the end-user about it.
**[Persona:](https://github.com/knative/eventing/blob/main/docs/personas.md)**
System Operator, Developer
**Exit Criteria**
The end-user could easily identify the Event Mesh configuration is invalid, and some messages are getting dropped. The best would be to allow use of well known K8s monitoring tools - using K8s events should be adequate.
**Time Estimate (optional):**
5d (events are dropped in number of places across the codebase)
**Additional context (optional)**
My proposal to solve this is to reconcile a Kubernetes event whenever such a situation occurs. Such an event may look like:
```yaml
apiVersion: events.k8s.io/v1
kind: Event
eventTime: 2025-04-23T09:09:54Z
metadata:
namespace: user-evening-bits-namespace
name: knative-eventing-mt-broker.1838e7822e31b835
labels:
eventing.knative.dev/event-type: my-event-type
eventing.knative.dev/event-source: my-event-source
regarding:
apiVersion: eventing.knative.dev/v1
kind: Broker
name: default
namespace: user-evening-bits-namespace
type: Warning
action: event-dropped
reason: EventLoop
note: Event of type "my-event-type" and source "my-event-source" has reached internal TTL, which most likely signals an event loop. The event was dropped.
series:
count: 351
lastObservedTime: 2025-04-23T09:09:54Z
```
Notice the `series.count`. It should be bumped whenever "same event" occurs again. In this case, the reconciler should match the K8s events using `metadata.labels` of `eventing.knative.dev/event-type` and `eventing.knative.dev/event-source`, and bump the `series.count` when the next message is being dropped.
Contributor guide
Research direction
Start by locating the event-drop paths and TTL handling described in the issue, including EventTransform-related loop prevention. Add a Kubernetes Warning event for dropped messages and ensure repeated matching events update series.count; verify that users can identify invalid configuration and dropped messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100