daeuniverse / daeuniverse/api-standardize

Proposal: connection lifecycle event stream (established/closed events with byte totals)

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
0
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Problem

`GET /api/connections` (current spec and PR #1's redesign) is a
pull-based snapshot. Two things a dashboard needs cannot be built from
it, no matter how fast it polls:

1. **Short-lived flows are invisible.** Connections that open and close
between two polls — plain DNS lookups, health checks, chatty apps —
never appear.
2. **Traffic history cannot be accurate.** "Bytes per site / process /
outbound / node over hours or days" needs each flow's *final* byte
totals, which only exist at close time and are never observed by
polling.

## Proposal

Add a push-based lifecycle event stream:

```
GET /api/connections/stream (SSE or WebSocket, capability-gated per #1)
```

- Events embed #1's connection object; a `kind` field distinguishes
`established` (flow opened) from `closed` (carries the flow's
*final* `upload_bytes` / `download_bytes`).
- Two attributes dashboards need most, beyond #1's current object:
`pname` and the selected node (dae's `outbound` names the routing
target — typically the group — not the node that carried the flow).
- **Backpressure**: the engine must never let a slow consumer stall the
data path — bounded queue with drop under pressure. Loss accounting
travels inside the stream: per-event sequence numbers (dropped events
leave a detectable gap) plus periodic `stats` events carrying a
cumulative `dropped` count and dropped-byte totals, so clients can
bound the error.

With this, a dashboard records every flow it observes, once, with exact
final totals — no polling losses, no diffing.

## Prior art

- I maintain a dae dashboard that consumes this kind of stream via a
small patch (one JSON event per datagram on a unix socket, exported
from the routing plane): it powers a 30-day traffic history with
drill-down by site / process / outbound / node. The routing plane
already knows each flow's metadata and byte totals at close time, so
the plumbing is cheap. The same instrumentation point would also
serve my routing-trace proposal (#2).
- Clash/mihomo's `WS /connections` — which honk already exposes via
`experimental.clash_api` — pushes periodic full-state snapshots.
Attribution exists for the flows a push happens to catch, but
between-push flows and their final totals are lost. Lifecycle events
fix this by pushing deltas instead of snapshots.

## Open questions

- Transport: SSE vs WebSocket (either way — browsers can set custom
headers on neither, so credential passing needs a decision).
- Baseline for flows already open at subscribe time (synthetic
`established` replay, or snapshot + dedupe by `id`).
- UDP session semantics: what `established` / `closed` mean without a
handshake (idle timeout).
- Sequence numbering: global or per-subscription (interacts with
server-side filtering, if we add it later — filtered-out events look
like gaps).
- Behavior across engine reloads/restarts (generation id on events?).

## Alternatives considered

- **Poll faster** — sampling losses never reach zero.
- **Push snapshots over WS (Clash-style, today's incumbent)** —
inherits the sampling gaps; only aggregate totals stay exact.
- **Pull-based changelog** (`GET .../events?since=`) — same
event source without the transport questions; same drop accounting.
- **Server-side aggregation endpoint** — moves history persistence
into the engine, a role change the current design avoids.
- **External observers** (pcap/conntrack/eBPF) — no `outbound`/node
attribution; that mapping lives inside the engine.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the proposed GET /api/connections/stream endpoint and the connection object from #1. Resolve the transport, subscription baseline, UDP semantics, sequence scope, and restart behavior before implementation. Done means the lifecycle event contract, backpressure and loss accounting behavior, and required attributes are specified consistently.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.