DataTalksClub / DataTalksClub/website

Catalog service-target queries, evidence, owners, and alert keys

Open
#266 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

decision enhancement human operations P1 testing
Dominant language
Python
Stars
0
Forks
0
No merged PRs in 30d

Description

Parent: #66

Product outcome

Provide one versioned, machine-validated source catalog that tells operators and later infrastructure work exactly how each approved service/recovery target and every release-critical failure is measured, classified, evidenced, owned, alerted, investigated, and reviewed.

This issue owns the declarative contract and its repository validation only. Domain applications own the meaning and production of their signals. DataTalksClub/aws-infra/#268 owns CloudWatch queries, alarms, dashboards, notification routing, and live readback. #269 owns operational exercises. A catalog row, synthetic fixture, or source test is never evidence that a live signal, alarm, dashboard, restore, or escalation works.

Normative authority

  • _docs/PROCESS.md: role-separated lifecycle, versioned verification, evidence classification, and sensitive-data rules.
  • _docs/specs/07-security-privacy-operations.md: observability inventory, alert ownership contract, approved target definitions, recovery, and release-critical failure behavior.
  • _docs/specs/08-aws-development-terraform.md: infrastructure-repository ownership for query-backed alarms, dashboards, destinations, backup metrics, and expiry monitoring.
  • _docs/specs/10-verification-strategy.md: boundary, missing/stale evidence, fault, restore, rollback, and alert verification.
  • Closed #26: owner approval of the initial target values. The approval explicitly leaves measurement, alarms, restore evidence, and named alert/runbook owners to #66.
  • #265: accepted redacted observability schema and bounded signal identifiers are a required input.

Dependencies and dispatch gate

Blocked for engineering until #265 is accepted. Every metric, event, or log source key must resolve to an accepted #265 identifier and schema version. Every domain-defined signal or receipt must link an accepted owning issue/contract. A catalog may not manufacture an event, reinterpret a generic AppEventCount, or treat /health/* or deployment smoke as availability, latency, RPO, or RTO.

The Relay rows also depend on #49's accepted intent/acceptance/reconciliation definitions. Backup/RPO and restore/RTO rows depend on the accepted provider-neutral backup receipt and later restore-duration receipt contracts under #66. Content, jobs, courses, events, API/security, deployment, and edge rows depend on their owning domain's accepted signal definitions. These are interface dependencies, not permission for #266 to implement those signals.

HUMAN owner, notification, escalation, review-cadence, and exception choices remain intentionally unresolved. They do not block PM grooming, but they block #266 closure and any claim that a row is operational.

Scope

1. Versioned catalog and schema

Add one canonical, reviewable data file under _docs/operations/ plus an offline validator and focused tests. The exact serialization format may follow an existing repository convention, but it must have deterministic ordering and a canonical digest. The catalog has a top-level catalog_version, generated_at is forbidden, and each row has one globally unique stable id.

Every row must contain the following keys exactly once; nullable HUMAN/domain fields remain present as null with a matching blocker rather than being omitted, guessed, filled with prose such as TBD, or represented by a wildcard:

Group Required keys and rules
Identity id, kind (service_target or release_critical_failure), environment (production, development, or both), authority, catalog_version
State readiness (blocked, source_defined, or closure_ready), blockers (issue/decision identifiers only), domain_owner (owning app/repository, not the accountable HUMAN)
Population scope, eligible_population, success_definition, failure_definition; no catch-all route, actor, account, tenant, country, raw URL, or unbounded identifier dimension
Source/query source_type (metric, event, log, receipt, or composite), source_contract, source_schema_version, source_key, query_language, query, unit, aggregation, grouping, clock_start, clock_end, ingestion_delay; a composite lists every constituent and exact join/window rule
Objective comparison, threshold, objective_window, evaluation_window, boundary_behavior; duration units are explicit and equality is tested
Exclusions exclusions, with an allowlisted stable classifier_key, authority, start/end clock, and effect for each exclusion; unrecorded maintenance, dependency failure, low traffic, or operator judgment is not an exclusion
Missing/stale data missing_data_state, missing_data_after, stale_data_after, missing_data_action; missing, partial, stale, unparsable, version-mismatched, or source/query disagreement is unknown/non-green and never inferred zero, success, or an exclusion
Evidence evidence_key, evidence_cadence, evidence_lookback, evidence_freshness, evidence_schema, evidence_redaction; source-only evidence and live evidence use distinct keys
Accountability accountable_owner, notification_destination, escalation_owner, escalation_destination; these are named roles/teams and approved bounded destinations, never issue labels or guessed repository maintainers
Operations links alert_key, runbook_key, dashboard_panel_key; keys are stable cross-repository references, not an AWS ARN, mutable console URL, secret, or proof that the resource exists
Actions/review breach_action, recovery_action, rollback_trigger, review_trigger, review_cadence, exception_owner, exception_expiry; exception expiry is mandatory when an exception exists

The validator rejects unknown keys/enums, missing/null fields without a declared blocker, duplicate IDs or keys, unresolved source/alert/runbook/panel references, unsafe/free-form dimensions, unbounded values, contradictory windows, invalid units/comparisons, missing-data success, exclusion without a machine classifier, owner expressed only as a label/issue, and sensitive-value canaries. Canonical output and digest are stable across repeated runs and input order changes.

2. Approved target rows

Record these approved #26 values without reopening or silently broadening them:

Stable row Approved objective Exact scope retained by the catalog
production.public_read.availability >= 99.9% per calendar month Eligible public read population; maintenance/dependency exclusions remain HUMAN unless explicitly approved and machine-classified.
production.registration_submission.availability >= 99.9% per calendar month Registration submission, separately queryable from reads and enrollment. Database rollback/validation denial is not silently counted as success.
production.enrollment_submission.availability >= 99.9% per calendar month Enrollment submission, separately queryable from reads and registration.
production.public_cached_response.p95 < 500 ms Cached public responses at the approved edge region under normal load. Exact route/population, edge region, load classifier, clock, and evaluation window require accepted domain/HUMAN input.
production.public_uncached_html.p95 < 1 s Uncached public HTML at the approved edge region under normal load, with the same unresolved classifier/clock inputs kept explicit.
production.relay_approved_intent.accepted_within_5m >= 99% accepted within 5 min Approved transactional intents only. Relay outage and suppression are excluded only through accepted classifiers. Provider acceptance is distinct from delivered, and ambiguous acknowledgement is not acceptance. The percentage evaluation window remains HUMAN.
production.github_content.freshness < 15 min From accepted main-branch commit to accepted active-content observation. GitHub failure serves last known good content but remains a freshness failure/alert unless an explicit owner-approved classifier says otherwise.
production.database.rpo <= 15 min Database recovery-point loss measured by an accepted backup/restore receipt; it does not imply coverage of assets, secrets, tombstones, queued intents, or Relay state. Exact reference clocks require the recovery contract.
production.service.rto <= 4 h Service recovery measured by an accepted drill receipt. Outage-start, restored-service end, required consumers/data/reconciliation, and abort handling require the recovery contract/HUMAN approval. Deployment compensation budgets are not RTO.
development.database.rpo <= 24 h Development database scope and accepted recovery receipt; whether this is target or risk acceptance remains recorded by the owner, not inferred.
development.service.rto <= 1 business day The catalog must reference an owner-approved business-day calendar/timezone and exact recovery clocks.

No development availability, latency, content-freshness, or Relay-acceptance value was approved by #26. Do not invent a value or mark one N/A; record their absence in the coverage report as human_decision_required only if the owning operations scope requires those rows.

3. Release-critical failure coverage

Create separately queryable stable rows—never one prose aggregate—for every spec-07 family below. Where a family contains multiple states, each state that changes threshold, owner, action, or runbook is a separate row:

  1. public, learner, Studio, and API availability/latency/error rate;
  2. registration/enrollment success, verification, throttling, and invariant failure;
  3. content freshness, failed/quarantined release, active-commit mismatch, link-build failure, and search-build failure;
  4. worker heartbeat, scheduled-job lateness, oldest durable-email-job age, replay/conflict, dead/ambiguous Relay projection, callback/reconciliation freshness, and queue depth;
  5. Relay submission availability/latency and Relay-supplied accepted, delivered, bounce, complaint, rejection, suppression, quota, and cost summaries, preserving accepted-versus-delivered semantics;
  6. course scoring failure, peer-assignment failure, and course deadline-job lateness;
  7. API authentication failure, authorization denial/failure classification, rate limiting, high-risk operation, and export volume;
  8. database/storage health, backup freshness/failure, restore-verification failure, edge/origin failure;
  9. bounded route/viewer/cache-status/cache-age, cache-hit ratio, origin-request rate, bytes/status, invalidation state/latency, WAF label/action/allowance, and edge-function errors;
  10. GitHub unavailable, invalid content quarantine, database transaction failure, worker/Relay unavailable or ambiguous, search/graph degradation, stale-revision conflict, scoring/registration/enrollment invariant conflict, deployment regression, immutable-identity mismatch, Relay credential/callback-secret expiry, and provider-health degradation.

Overlap is resolved by a shared source_key and explicit row references, not duplicate definitions. Security denials, expected validation failures, and throttles must remain distinguishable from system failures. No row contains PII, request/response bodies, complete queries, raw IP/header/country suggestion, email, profile/submission content, provider payload, token, credential, secret, database row/export, high-cardinality object ID, or mutable image tag.

4. Deterministic query and evidence behavior
  • A closure_ready row has a non-null exact query, source contract/version/key, clocks, unit, aggregation, scope, objective/evaluation window, exclusions, missing/stale behavior, evidence contract, owners/destinations, action triggers, and stable alert/runbook/panel keys.
  • Queries must be executable in their declared language by the later owning adapter. Pseudocode, prose, console-only saved searches, moving URLs, and * are invalid.
  • Healthy, exact-boundary, breach, recovery, excluded, missing, partial, stale, version-drift, duplicate, and query/panel-disagreement fixtures produce deterministic dispositions. Equality follows each row's declared comparison; it is never rounded into success.
  • Evidence records bind catalog version/digest, query/source versions, environment, bounded observation window, result/disposition, and immutable release identity when application code is involved. They contain no credentials, provider payloads, production records, or sensitive dimensions.
  • missing_data_state is always non-green. A row may page, ticket, block release, or require investigation only when that action has an accepted owner/destination; otherwise it remains blocked and reports the missing decision.
  • The catalog declares cadence and freshness but does not schedule collection. Source-only validation is not live evidence and must say so in every report.

Source-only delivery slice

After #265 is accepted, engineering may deliver a bounded repository-only slice containing:

  • the schema, canonical catalog format/digest, validator, and safe fixture vocabulary;
  • the complete approved-target and release-critical inventory above;
  • exact references for already accepted domain sources and explicit null + blockers for every unresolved query, classifier, clock, owner, destination, cadence, action, or cross-repository key;
  • offline tests for schema, coverage, canonicalization, boundary/missing/stale/exclusion behavior, reference integrity, and sensitive/cardinality canaries;
  • a machine-readable coverage report that distinguishes blocked, source_defined, and closure_ready without treating blocked rows as passing.

This first slice uses Refs #266, leaves the issue open with human/decision, and makes no operational-readiness claim. It requires normal engineer, independent tester, PM, commit/merge/push, and on-call gates for its own repository artifact. Browser tests and screenshots are not_applicable because this issue adds no product/operator UI; the versioned verification plan must still justify that disposition.

Closure gates

#266 closes only after a later lifecycle pass updates the accepted source catalog so that:

  • every row is closure_ready; #265 and every owning-domain signal/receipt contract is accepted and referenced;
  • the accountable HUMAN approves each owner, notification destination, escalation path/destination, review cadence, exception policy/expiry, exclusion classifier, clock interpretation, and rollback/review trigger;
  • every query is exact and repository-valid against accepted synthetic/contract fixtures, with healthy, boundary, breach, recovery, excluded, missing, stale, and drift cases independently tested;
  • every alert/runbook/dashboard-panel key is unique, stable, cross-repository resolvable, and ready for #268 consumption, without claiming the AWS resources exist;
  • the final coverage report has zero missing, duplicate, wildcard, unresolved, or blocked rows and binds the exact catalog digest;
  • independent tester PASS and PM acceptance cover the frozen final candidate.

Live AWS query execution, alarm firing/recovery, dashboard agreement/readback, notification delivery, backup/restore measurement, and rehearsal evidence remain mandatory downstream #268/#269 and parent-#66 gates; they are explicit not_implemented_here, never closure evidence for this source-only issue.

Acceptance criteria

  • The versioned catalog/schema includes every required key and rejects every malformed, ambiguous, unsafe, unbounded, duplicate, unresolved, or missing-data-as-success case above.
  • All 11 approved target rows preserve the exact #26 values/scopes; no unapproved development target, exclusion, owner, evaluation window, clock, or business-day interpretation is invented.
  • Every release-critical spec-07 family/state has one stable, separately queryable row with explicit domain ownership and no sensitive/high-cardinality dimension.
  • Every accepted source/query is exact, versioned, deterministic, executable by its declared later adapter, and backed by boundary/exclusion/missing/stale/drift fixtures; unresolved inputs remain null with issue/decision blockers.
  • Missing, partial, stale, unparsable, mismatched, absent-owner, absent-destination, and query/panel-disagreement states are non-green and cannot disappear through an exclusion or aggregate.
  • Evidence binds catalog/source/query versions and digest, environment, observation window, disposition, and applicable immutable release identity while remaining redacted and synthetic/repository-safe.
  • The source-only slice reports its limitations, uses Refs #266, keeps the issue open, and contains no AWS/provider mutation, alarm/dashboard implementation, scheduled collection, restore/drill, or UI change.
  • [HUMAN] Accountable owners approve every row's owner/destination/escalation, exclusions, clocks/windows, cadence, breach/recovery/rollback/review action, and exception handling/expiry.
  • Before closure, every row is closure_ready, all references resolve to accepted contracts, the coverage report is zero-gap, and separate engineer/tester/PM evidence passes on one frozen candidate.

Required repository scenarios

  1. Load the canonical complete catalog; validate deterministic serialization/digest and all references.
  2. Reject missing/unknown key, unknown enum/unit/comparison, duplicate ID/reference, contradictory clock/window, invalid threshold, wildcard/free-form query, unbounded dimension, and owner supplied only as a GitHub label/issue.
  3. For each approved objective, test healthy, exact equality, just-breached, recovered, excluded, missing, partial, stale, and source/query-version drift according to the declared comparison.
  4. Relay fixtures distinguish approved/unapproved, accepted/delivered/ambiguous, outage, and suppression; an unclassified outage or suppression is not excluded.
  5. Availability separates public reads, registration submissions, and enrollment submissions; expected denial/throttle/validation and invariant/system failure remain distinct.
  6. Cached/uncached latency requires accepted cache/load/region classifiers; content freshness binds accepted-main and active-content clocks; RPO/RTO require accepted receipts and reject deployment-compensation timing.
  7. Sensitive/cardinality canaries cover all forbidden values and raw identifiers; no invalid value appears in validation errors or artifacts.
  8. Source-only coverage deliberately retains unresolved HUMAN/domain blockers and cannot emit closure_ready; final closure fixtures resolve all rows with no implicit defaults.

Explicit non-goals

  • No #265 event/schema implementation and no domain metric/event/receipt production or semantic change.
  • No CloudWatch Logs Insights/metric-math resource, alarm, dashboard, notification topic/destination, AWS Budget, Terraform, IAM, backup, restore, provider call, credential inspection, or production/development environment mutation.
  • No inference of owners from contributors, labels, code ownership, issue assignment, repository access, or incident history; no inference of destinations, exclusions, maintenance windows, business calendars, evaluation windows, or escalation policy.
  • No reopening the approved #26 target values and no invented development availability/latency/freshness/Relay objective.
  • No claim that health/readiness, deployment smoke, generic event counts, a source validator, or synthetic fixtures prove live SLOs, alarms, dashboards, RPO/RTO, or operational readiness.
  • No raw/sensitive/high-cardinality operational data in source, tests, logs, issues, screenshots, or reports.
  • No product page, Studio surface, API, template, CSS, JavaScript, copy, or browser redesign.

Lifecycle handoff

PM grooming makes the contract explicit but does not clear its dependency. Leave engineering undispatched until #265 is accepted and the first usable domain source contracts exist. The source-only candidate then follows the complete PROCESS lifecycle and lands with Refs #266. HUMAN/domain gaps are filled only through a later reviewed update; after zero-gap tester PASS and PM acceptance, that focused commit may use Closes #266. #268 consumes the accepted final catalog and owns all infrastructure/live evidence.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read _docs/PROCESS.md and the referenced security, infrastructure, and verification specifications first, then inspect existing conventions under _docs/operations/. Engineering is blocked until #265 is accepted and several owner decisions are resolved. Done means a versioned catalog, offline validator, focused tests, and deterministic validation behavior covering the stated contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.