[v2] Restore CloudEvents egress for run/action lifecycle (Kafka / external sinks)
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 120
Description
## Summary
In Flyte 1 we rely on Admin CloudEvents egress to Kafka so downstream systems can react to workflow lifecycle without polling. Flyte 2 appears to replace that with gRPC watch streams (`WatchRuns` / `WatchActions`) and durable `action_events` in Postgres, but there is no first-class CloudEvents → Kafka (or other sink) publisher yet.
## Flyte 1 behavior (what we have today)
- Admin publishes CloudEvents to Kafka (`cloudEvents.eventsPublisher`, e.g. topic `flyte.admin.cloudevent`).
- Downstream consumers key off types like:
- `com.flyte.resource.flyteidl.admin.WorkflowExecutionEventRequest`
- (also available) `NodeExecutionEventRequest` / `TaskExecutionEventRequest`
## Flyte 2 gap
- Runtime model is **run / action** (`ActionEvent`), not workflow / node / task graph events.
- External visibility today is primarily:
- gRPC `WatchRuns` / `WatchActions` (live; Postgres LISTEN/NOTIFY wake-ups)
- durable `action_events` table (no pub/sub egress)
- `flyteidl2/event/cloudevents.proto` still carries v1 CloudEvent shapes; there does not appear to be an equivalent ActionEvent/Run CloudEvents publisher for v2.
Without egress, each operator must build a bridge: watch (or poll) → republish to Kafka/SNS/etc., including reconnect/catch-up semantics.
## Ask
1. **Feature:** First-class CloudEvents (or equivalent) egress for Flyte 2 run/action lifecycle to external sinks (Kafka at minimum; ideally pluggable like v1).
2. **Docs / migration guide:** How to replace Admin CloudEvents consumers for the common pattern “on run SUCCEEDED/FAILED, notify external systems,” including:
- recommended API (`WatchRuns` vs polling `action_events` / `ListRuns`)
- event schema mapping (v1 `WorkflowExecutionEvent` → v2 `Run` / `ActionEvent`)
- durability, replay, and at-least-once guidance
- whether node/task-level CloudEvents have a v2 counterpart or are intentionally dropped
## Motivation
Teams with event-driven orchestration need either a native sink or an official migration path so they are not inventing a one-off bridge per deployment.
## Acceptance criteria (suggested)
- [ ] Documented migration path from v1 Admin CloudEvents → v2
- [ ] Either: configurable publisher for run (and optionally action) lifecycle to Kafka / CloudEvents, **or** an official reference “watch → Kafka” bridge
- [ ] Clear event type / payload contract for terminal run phases (SUCCEEDED / FAILED / ABORTED equivalents)
- [ ] Notes on idempotency keys for consumers (e.g. run/action/attempt/phase/version)
Contributor guide
Assessment
This issue has not been assessed yet.