AbsaOSS / AbsaOSS/EventGate

Propagate the request correlation id to downstream sinks

Offen
#212 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Python
Sterne
4
Forks
0
Ø Merge
1 T. 5 Std.
Gemergte PRs (30 T.)
8

Beschreibung

### Feature Description

Propagate the request correlation id to the downstream sinks, so consumers of the fan-out targets can join their processing logs to the EventGate request that produced the message:

- **Kafka** — add the id as a record header (e.g. `correlation_id`) in `WriterKafka.write()`. Record headers keep the payload untouched, and consumers can read the header without schema changes.
- **EventBridge** — include the id in the `put_events` entry, either as a top-level field of `Detail` or via `TraceHeader`, in `WriterEventBridge.write()`.
- **Postgres** — optional; would require a schema change (extra column), so likely out of scope for the first iteration.

### Problem / Opportunity

Since #193 / PR #204, every EventGate request resolves a correlation id (`X-Correlation-ID` → `X-Request-ID` → API Gateway request id), logs it on every line and returns it in the `X-Correlation-ID` response header. The id currently stops at EventGate: once a message is fanned out, downstream consumers have no way to correlate their processing with the originating request.

Beneficiaries: teams consuming the Kafka topics and EventBridge events, and anyone debugging a cross-system flow end-to-end — one id would then trace a message from the caller, through EventGate's logs, into every sink's consumer logs.

### Acceptance Criteria

1. A message published to Kafka carries the request's correlation id as a record header; the message payload (value) is byte-for-byte unchanged.
2. An event published to EventBridge carries the request's correlation id in its entry metadata (`TraceHeader` or a documented `Detail` field); the schema-validated payload is unchanged.
3. When no correlation id is resolved (empty id), no header/field is added and writes behave exactly as today.
4. Existing topic schema validation and all existing consumers are unaffected — propagation uses transport metadata only, never the message body.
5. Unit tests cover both writers with and without a bound correlation id; integration tests prove the header/field arrives in the sink.
6. README "Logging & Correlation" documents the downstream propagation contract.

### Proposed Solution

The writers do not currently receive the correlation id; it is bound in `src/utils/observability.py`. Two options:

- a small accessor (e.g. `current_correlation_id()`) in `src/utils/observability.py`, read by the writers, or
- an explicit parameter threaded through `HandlerTopic._write_to_all()` into `Writer.write()`.

The explicit parameter is more testable and keeps writers free of hidden state; the accessor avoids touching the `Writer` interface. Alternative considered and rejected: putting the id into the message body — breaks topic schema validation and changes consumer contracts.

The id is already validated against `^[A-Za-z0-9._:-]{1,128}$` before use, so it is safe to forward verbatim.

### Dependencies / Related

- #193 (structured logging + correlation id)
- PR #204 (implementation; ADR `adr/001-observability/001-observability.md`)

### Additional Context

Follow-up proposed during the second-opinion review of PR #204 (item P7). Postgres propagation can be a separate issue if a schema change is ever justified.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.