events: CloudEvents egress and broker sinks
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 20
- Forks
- 2
- Avg merge
- 5h 10m
- Merged PRs (30d)
- 127
Description
Job lifecycle events exist inside FlexiQ — the scheduler produces ResultOutcome, the SDK loop
dispatches hooks and events per outcome, and there is a pub/sub mechanism in the core. All of it
is reachable only from inside a process running FlexiQ code, in one of three languages.
So anything that wants to react to job events — a data warehouse, an alerting rule, a workflow
engine, an internal dashboard that is not this dashboard — has to poll the database or embed an
SDK. Both are worse than the thing that already exists internally.
Add event egress:
- CloudEvents over HTTP as the default shape. It is the interoperable spec, and every
eventing platform reads it. - Broker sinks — Kafka, NATS, Redis Streams. Redis is already a dependency for one backend,
so it is the cheapest to ship first. - Filterable by namespace, queue, task and event type. Nobody wants every transition of
every job.
Requirements:
- Delivery semantics stated, not implied. At-least-once with a documented dedupe key, or
at-most-once and say so. Silence here means every consumer assumes exactly-once and is wrong. - A slow or dead sink must not back-pressure the scheduler. Bounded buffer, drop with a
counter, and the drop visible in metrics. The shedding machinery already models "dropping is a
legitimate outcome that must be observable". - No payloads by default. A job's arguments can be personal data. Emit metadata; make
payload inclusion opt-in per sink and loud about it. - The same SSRF constraints as push dispatch (#843) apply to an HTTP sink.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing ResultOutcome, the SDK loop's outcome hooks, the core pub/sub mechanism, and the shedding machinery described in the issue. Read push dispatch in #843 for the HTTP sink's SSRF constraints; the design is done when delivery semantics, filtering, bounded shedding and metrics, metadata-only defaults, and opt-in payload handling are specified for each sink.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, redis, rust
- Domain
- api, backend, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100