ByteVeda / ByteVeda/flexiq

events: CloudEvents egress and broker sinks

Open
#848 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/server enhancement P2
Dominant language
Rust
Stars
20
Forks
2
Avg merge
5h 10m
Merged PRs (30d)
127

Description

Job lifecycle events exist inside FlexiQ — the scheduler produces ResultOutcome, the SDK loop
dispatches hooks and events per outcome, and there is a pub/sub mechanism in the core. All of it
is reachable only from inside a process running FlexiQ code, in one of three languages.

So anything that wants to react to job events — a data warehouse, an alerting rule, a workflow
engine, an internal dashboard that is not this dashboard — has to poll the database or embed an
SDK. Both are worse than the thing that already exists internally.

Add event egress:

  • CloudEvents over HTTP as the default shape. It is the interoperable spec, and every
    eventing platform reads it.
  • Broker sinks — Kafka, NATS, Redis Streams. Redis is already a dependency for one backend,
    so it is the cheapest to ship first.
  • Filterable by namespace, queue, task and event type. Nobody wants every transition of
    every job.

Requirements:

  • Delivery semantics stated, not implied. At-least-once with a documented dedupe key, or
    at-most-once and say so. Silence here means every consumer assumes exactly-once and is wrong.
  • A slow or dead sink must not back-pressure the scheduler. Bounded buffer, drop with a
    counter, and the drop visible in metrics. The shedding machinery already models "dropping is a
    legitimate outcome that must be observable".
  • No payloads by default. A job's arguments can be personal data. Emit metadata; make
    payload inclusion opt-in per sink and loud about it.
  • The same SSRF constraints as push dispatch (#843) apply to an HTTP sink.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing ResultOutcome, the SDK loop's outcome hooks, the core pub/sub mechanism, and the shedding machinery described in the issue. Read push dispatch in #843 for the HTTP sink's SSRF constraints; the design is done when delivery semantics, filtering, bounded shedding and metrics, metadata-only defaults, and opt-in payload handling are specified for each sink.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, redis, rust
Domain
api, backend, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.