knative / knative/eventing

Inform user of dropped events

Open
#8,572 18 comments 1 reaction 0 assignees View on GitHub
good first issue kind/feature-request triage/accepted
Dominant language
Go
Stars
1.6k
Forks
631
Avg merge
2d 1h
Merged PRs (30d)
6

Description

**Problem**
Whenever events are being dropped by event mesh, we should inform the end-user about that. Just logging is not enough for such a case. IMHO, the best to create a dedicated warning level Kubernetes event, which is designed for such a case: https://www.cncf.io/blog/2023/03/13/how-to-use-kubernetes-events-for-effective-alerting-and-monitoring/

**Background**
With the recent addition of the EventTransform API, the chance of misconfiguration by the end-user raises greatly, as they could easily create infinite loops by transforming the source event in a way the original trigger matches again. The TTL mechanism should eventually break that loop, but in that case, we should inform the end-user about it.

**[Persona:](https://github.com/knative/eventing/blob/main/docs/personas.md)**
System Operator, Developer

**Exit Criteria**
The end-user could easily identify the Event Mesh configuration is invalid, and some messages are getting dropped. The best would be to allow use of well known K8s monitoring tools - using K8s events should be adequate.

**Time Estimate (optional):**
5d (events are dropped in number of places across the codebase)

**Additional context (optional)**
My proposal to solve this is to reconcile a Kubernetes event whenever such a situation occurs. Such an event may look like:

```yaml
apiVersion: events.k8s.io/v1
kind: Event
eventTime: 2025-04-23T09:09:54Z
metadata:
namespace: user-evening-bits-namespace
name: knative-eventing-mt-broker.1838e7822e31b835
labels:
eventing.knative.dev/event-type: my-event-type
eventing.knative.dev/event-source: my-event-source
regarding:
apiVersion: eventing.knative.dev/v1
kind: Broker
name: default
namespace: user-evening-bits-namespace
type: Warning
action: event-dropped
reason: EventLoop
note: Event of type "my-event-type" and source "my-event-source" has reached internal TTL, which most likely signals an event loop. The event was dropped.
series:
count: 351
lastObservedTime: 2025-04-23T09:09:54Z
```
Notice the `series.count`. It should be bumped whenever "same event" occurs again. In this case, the reconciler should match the K8s events using `metadata.labels` of `eventing.knative.dev/event-type` and `eventing.knative.dev/event-source`, and bump the `series.count` when the next message is being dropped.

Contributor guide

Open the contributing guide

Research direction

Start by locating the event-drop paths and TTL handling described in the issue, including EventTransform-related loop prevention. Add a Kubernetes Warning event for dropped messages and ensure repeated matching events update series.count; verify that users can identify invalid configuration and dropped messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.