ADORSYS-GIS / ADORSYS-GIS/lightbridge-authz
[Ticket]: Generic replay job + promotion rehearsal for the raw OTLP archive
- Ngôn ngữ chính
- Rust
- Star
- 0
- Fork
- 1
- Merge trung bình
- 6 giờ 42 phút
- Pull request đã merge (30 ngày)
- 246
Mô tả
### Type
Feature
### Summary
We need a single generic replay job that reads any slice of the raw OTLP archive (S3, prefixed by trusted source + date) and re-feeds it through the real authenticated ingest endpoint, so promoting a field to a column comes with historical backfill for every source — and a re-run changes no counts.
Expected result: one source-agnostic replay job plus a demonstrated end-to-end promotion rehearsal (add a nullable column → replay a historical window → the column backfills → counts unchanged).
### Intent
The archive leg (#589) is only half the loop. This ticket is the **replay** half: a single source-agnostic job that reads an archived prefix/date range and POSTs it through the real ingest endpoint, made idempotent by the grain-table dedup keys, plus a demonstrated end-to-end promotion rehearsal.
### Source of truth (links)
- https://github.com/ADORSYS-GIS/lightbridge-authz/issues/589 (parent story — raw OTLP archive on the collector)
- https://github.com/ADORSYS-GIS/lightbridge-authz/issues/581 (epic — multi-source usage ingestion)
- https://github.com/ADORSYS-GIS/lightbridge-authz/issues/582 (execution grain — dedup keys)
- https://github.com/ADORSYS-GIS/lightbridge-authz/issues/583 (day/seat grain — dedup keys)
- `docs/plans/0581-multi-source-usage-plan-of-work.md` (PR-2b)
### Current Behavior
The current `usage_events` table has **no dedup key** (only `id BIGSERIAL PRIMARY KEY`; the insert in `crates/lightbridge-authz-usage/src/repo.rs` is a plain `INSERT … VALUES` with no `ON CONFLICT`) and **no `source` column**. Idempotent replay and lossless source round-trip are therefore impossible against the current table — this ticket is blocked on the grain tables that add both.
### Expected Behavior
A single generic replay job (source-agnostic: no per-vendor code) reads an archive prefix/date range and POSTs it through the real authenticated ingest endpoint; grain-table dedup keys make any replay idempotent — a re-run changes no counts. A promotion rehearsal is demonstrated end to end.
### Acceptance Criteria
- [ ] A single generic replay job (source-agnostic: no per-vendor code) reads an archive prefix/date range and POSTs it through the real authenticated ingest endpoint; grain-table dedup keys make any replay idempotent — a re-run changes no counts (test proves it).
- [ ] A promotion rehearsal is demonstrated end to end: add a nullable column, replay a historical window, the column backfills; counts unchanged.
- [ ] Error cases are handled safely (archive read failure, ingest failure) — fail loud, never silently drop.
- [ ] Existing behavior is not broken.
- [ ] Relevant tests are added or updated.
- [ ] Verification evidence is provided.
### Out of Scope
- The archive leg itself (separate ticket — the S3 exporter on the edge collector).
- Historical Copilot data (never transited the collector — #588's one-time governance-ctl S3-NDJSON backfill is the only source for it).
- Replaying from Mimir/Loki/Tempo (rejected: lossy/short-retention/bespoke-extraction).
- Any per-tool archive code.
### Technical Context
The replay job naturally lives beside the usage service it feeds (`crates/lightbridge-authz-usage`). It is a small binary: list objects under an S3 prefix/date range, POST each through the real ingest endpoint, and rely on the grain dedup keys for idempotency. Bucket prefix layout `///
### Risks
- **Blocked on #582 / #583** (grain tables with dedup keys + `source` dimension) — this ticket cannot land before them.
- **D9 / D10 / D11** owner decisions gate the format and the failure/retention posture — settle before estimating, not during.
### Test Plan
- Round-trip test (archive → replay → identical grain rows).
- Idempotent re-replay test (replay twice → counts unchanged).
- A rehearsed promotion backfill in CI or a documented runbook run.
### Verification evidence
To be filled at completion with real command output.
### Human accountable owner
@stephane-segning
### AI Usage Declaration
- [x] Drafting the ticket
- [x] Understanding code
- [x] Proposing implementation
### Human verification completed
- [ ] I understood the intent
- [ ] I checked the source of truth
- [ ] I reviewed all AI-generated text/code
- [ ] I verified the implementation manually
- [ ] I verified the tests
- [ ] I checked for hallucinated assumptions
- [ ] I documented remaining risks
- [ ] I am the accountable owner and accept responsibility for this ticket.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.