DataTalksClub / DataTalksClub/website
Expose direct-sync source management in Studio and admin API
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Parent epic: #38
Management foundations: #32 and #33
Direct-sync storage/runner/operations: #273, #274, and #275
Product outcome
Expose the owner-approved direct-sync authority as one capability-scoped management product in Studio and /api/v1/admin/: operators can inspect configured sources, truthful current status, immutable run history, freshness, and redacted diagnostics; open the exact source location in GitHub for edits; request a sync of the configured current branch head; safely retry an eligible failed/partial immutable run; and, with the stronger configuration permission, change only the frozen allowlisted source policy.
Studio and the admin API are adapters over the same content/content_sync query and command services. Neither adapter directly mutates models, calls GitHub, parses content, or dispatches an unfenced job. GitHub remains read-only from the website. The new surface has no candidate preview, ready, activate, rollback, force-SHA, arbitrary repository, or arbitrary branch action.
PM disposition — blocked contract / do not dispatch
Keep needs grooming. The product routes, capability split, visible states, command semantics, security boundaries, and verification contract are fixed below, but an engineer-safe handoff still lacks accepted predecessor interfaces:
- #32/#33 must land the production role, reauthentication/high-risk, API-proof, credential, and management-parity policy used by source configuration, manual sync, and retry. #28 approved explicit “are you sure” confirmation plus reauthentication and deferred dual approval, but #32/#33 still own the concrete production hooks/symbols and API equivalent.
- #273 must land the exact
ContentSource,ContentSyncLog, revision, status, diagnostics, configuration, recovery-link, and immutable-history fields plus content-owned query/transition services. - #274 must land the exact direct-upsert runner result/recovery taxonomy and the eligibility rule for retrying an immutable failed/partial input.
- #275 must land the provider-neutral manual/reconciliation request seam, durable-operation/job identity, source policy, freshness calculation, safe diagnostics/reason catalog, checkout/current-head authorization rule, and exact 14-minute schedule/alert budget.
- The owner-approved first-source/rollout policy must freeze which configuration fields are operator-editable, which are code/configuration-owned, field masking, source object policy, and the effect of a configuration revision on current evidence and follow-up scheduling.
No engineer may infer a model or service name, enable a source, invent a GitHub/credential policy, expose a secret reference, or implement a UI over current staged ContentRelease activation services. After the dependencies are accepted, PM must replace every predecessor placeholder below with the exact accepted symbols/policy identifiers, re-audit routes and fields against current main, then remove needs grooming and post ## Grooming Complete.
Normative authority
_docs/PROCESS.md: issue lifecycle, dependency ordering, independent tester/PM gates, screenshots, versioned verification, and no engineering whileneeds groomingremains._docs/specs/open-decisions.mddecision 1 and #226: source lock → immutable checkout → parse/dispatch → direct upsert → truthfulSyncLog; no ordinary candidate, activation pointer, or product rollback graph._docs/specs/01-platform-architecture.md,_docs/architecture/app-boundaries.md, and_docs/architecture/shared-primitives.md: one Django deployment, business mutations in shared application services, optimistic revision/idempotency/audit in the command, and provider/job effects only after commit._docs/specs/03-github-content-and-people.md: source ownership, provenance, edit-on-GitHub, safe diagnostics, and no database editorial override remain normative. Candidate preview/activate/rollback text is superseded where it conflicts with #226/#38._docs/specs/06-studio-and-admin-api.md: one capability registry, Studio/admin API route-service-permission-result parity, UUID management resources,If-Match,Idempotency-Key, safe errors, bounded pagination, OpenAPI 3.1, private/no-store/noindex, and explicit confirmation for high-risk work. Release/activation/rollback entries are superseded for direct-sync authority._docs/specs/07-security-privacy-operations.md: deny by default, no object-existence leaks, below-15-minute content freshness, last-known-safe reads, redacted diagnostics, accessibility, monitoring, and no secrets/PII/provider payloads._docs/specs/10-verification-strategy.md: capability/OpenAPI parity, revision/idempotency/concurrency/fault coverage, Studio browser coverage, accessibility, and complete evidence.- #38 defines Slice E and the canonical direct-sync status/recovery boundary. #273–#275 own storage, execution, ingress/provider/jobs, reconciliation, and freshness; this issue adds only management adapters and their shared management-facing services where an accepted predecessor does not already supply one.
Exact capability and role contract
Declare each capability once in the accepted management registry, with the exact Studio/API routes below, one shared service, object policy, field policy, concurrency/idempotency policy, audit action, confirmation/reauthentication policy, and parity factory:
| Capability key | Purpose | Effective human roles |
|---|---|---|
content.source.view |
List/detail safe source status, freshness, and run history | site_admin, content_operator, auditor |
content.source.diagnostics.view |
Read bounded redacted run diagnostics | site_admin, content_operator, auditor with the stricter field mask below |
content.source.edit_link.view |
Reveal an allowlisted GitHub edit/source link | site_admin, content_operator |
content.source.configure |
Change the accepted mutable subset of one existing source policy | site_admin only unless the accepted #32 role matrix explicitly grants a narrower object policy |
content.source.sync |
Request observation/sync of one source's configured current branch head | site_admin, content_operator |
content.sync_run.retry |
Create a new recovery run for one eligible immutable failed/partial run | site_admin, content_operator |
API principals require the same exact capabilities/scopes and object/field policy. A bearer token, is_staff, group name, superuser flag, knowledge of a UUID, or access to one source never grants another capability. support_operator, course_operator, event_operator, and email_operator receive no capability here by default. Authorized querysets apply before lookup so denied and unknown UUIDs have the accepted indistinguishable response. Inactive/offboarded humans and disabled/revoked/expired principals fail closed immediately.
The final pass must bind these keys to #32/#33's accepted role and high-risk policy symbols. If predecessor naming or policy makes a key incompatible, PM updates this table explicitly; engineering may not silently rename or collapse capabilities.
Exact Studio routes
All routes are authenticated, private/no-store/noindex, CSRF-protected on POST, absent from sitemap/search, and use the ordinary Studio shell and navigation.
GET /studio/content/sources/— bounded/filterable source list.GET /studio/content/sources/<uuid>/— one source overview with safe configuration summary, current status, freshness, and recent runs.GET /studio/content/sources/<uuid>/sync-runs/— paginated immutable run history.GET /studio/content/sync-runs/<uuid>/— one run result and bounded evidence summary.GET /studio/content/sync-runs/<uuid>/diagnostics/— permission/field-policy-filtered diagnostics; never a raw log viewer.GET/POST /studio/content/sources/<uuid>/configuration/— view and submit the accepted editable configuration subset with expected revision.POST /studio/content/sources/<uuid>/sync/— confirmed manual current-head observation/sync request.POST /studio/content/sync-runs/<uuid>/retry/— confirmed eligible recovery request creating a new linked run.
A successful Studio POST redirects to the canonical source or new operation/run detail and renders an accessible status message. Refresh/back navigation cannot repeat a command. POST forms carry server-generated idempotency identity, expected revision, bounded reason, confirmation binding, and the final #32 reauthentication evidence; user input never carries a repository URL, branch, ref, SHA, adapter, path, credential, secret, job name, or provider payload except through the separately authorized configuration form's exact allowlist.
There is no Studio route for candidate preview, prepare, ready, activate, rollback, force sync, arbitrary SHA, source creation/deletion, raw payload/log download, repository browsing, credential display, or provider mutation. Existing historical compatibility routes, if still present before Slice F, are not linked or registered as actions for a direct-sync source and are not modified by this issue.
Exact admin API routes and operation IDs
All responses are JSON, Bearer-authenticated, private/no-store, CORS-denied by default, UUID-addressed, and represented in the generated OpenAPI 3.1 document.
| Method and route | OpenAPI operation ID |
|---|---|
GET /api/v1/admin/content-sources |
listContentSources |
GET /api/v1/admin/content-sources/<uuid> |
retrieveContentSource |
PATCH /api/v1/admin/content-sources/<uuid> |
updateContentSourceConfiguration |
GET /api/v1/admin/content-sources/<uuid>/sync-runs |
listContentSourceSyncRuns |
POST /api/v1/admin/content-sources/<uuid>/sync |
requestContentSourceSync |
GET /api/v1/admin/content-sync-runs/<uuid> |
retrieveContentSyncRun |
GET /api/v1/admin/content-sync-runs/<uuid>/diagnostics |
retrieveContentSyncRunDiagnostics |
POST /api/v1/admin/content-sync-runs/<uuid>/retry |
retryContentSyncRun |
PATCH requires If-Match for the source revision. Both command POSTs require Idempotency-Key, the accepted explicit confirmation representation, bounded reason, and the final #33 API reauthentication proof when their registered high-risk policy requires it. Missing preconditions fail safely; stale revision returns 409; exact replay returns the original safe result; same key with changed canonical intent conflicts; denied and unknown objects do not leak existence.
Long-running sync/retry returns 202 Accepted with one operation resource and the safe linked source/run identity supplied by #275. It never waits for checkout/provider/parser work. Configuration returns the updated safe source representation only after its shared command commits. The final pass must freeze exact request/response schema names and status/error codes against #33/#273–#275 rather than adding a parallel operation or error envelope.
Read model, status, freshness, history, and diagnostics
Studio and API serialize the same service DTOs and ordering. Templates and serializers never receive raw ORM models, provider values, or arbitrary diagnostic mappings.
Source list/detail safe fields
The common safe representation contains only the final accepted names for:
- source UUID and bounded operator label/stable key;
- enabled state and optimistic revision;
- adapter/ownership-policy identifiers allowed by the field policy;
- exact freshness target (14 minutes for the accepted first-source contract), derived freshness state, safe age/overdue duration, and evaluation timestamp;
- current
last_sync_status, immutable latest-run UUID, pending-follow-up boolean, and safe lock/queue age/state without owner/fence tokens; - last attempted, last successful, and last authenticated remote-head-observation timestamps plus commit identities/digests only where the final source field policy permits them;
- accepted direct-sync/adapter/parser/render/selection/source-status policy versions or digests where useful for diagnosis; and
- bounded recent-run counts by safe state.
Freshness is orthogonal to run status. It must use #275's accepted local-clock calculation and #38 mapping: only current complete evidence (success or verified unchanged-head skipped) may be fresh; queued/running are in progress; partial, failed, stale, unavailable, conflicting, coalesced skip, missing evidence, pending follow-up, and contradictory current evidence are never displayed as healthy/complete merely because an older success exists. Provider commit timestamps do not reset freshness.
List defaults to stable source ordering and bounded pagination. Exact allowlisted filters are status, freshness state, enabled state, and adapter/ownership family; exact sort allowlist is operator label/stable key, freshness age, last attempt, last success, and current status. Unknown filters/sorts fail with the normal safe field error. Empty list and no-filter-results are distinct accessible states.
Immutable run history/detail
History is newest requested time then UUID, with bounded pagination and safe filters for terminal/current status, trigger, reason code, and time window. A run representation may include only its UUID, source UUID, status, trigger, safe reason code, request/start/finish timestamps, eligible-for-retry boolean plus bounded ineligibility code, recovery predecessor/successor UUIDs, bounded created/updated/unchanged/drafted/rejected and partition counts, accepted state/evidence/version digests, and commit/tree identity only where the source field policy permits it. Terminal rows are never editable or reopened.
Diagnostics
Diagnostics are an allowlisted structured projection from #273–#275, not exception text or source content. They may expose bounded reason codes, counts, durations/ages, policy/version/evidence digests, and normalized bounded source-relative field/path pointers only to site_admin/content_operator when the final field policy allows them. The auditor view omits edit links, repository/ref details, commit/tree values if restricted, and source-relative pointers; it retains status, safe reasons, counts, timestamps, and evidence digests needed for audit review.
Every response, page, log, audit, metric, screenshot, operation, and OpenAPI example excludes raw webhook body/signature/delivery identity/header/payload; secret or credential reference/value; authorization/cookie/session/CSRF data; repository/provider response; checkout/absolute/temp path; source Markdown/YAML/HTML/body; rendered private data; exception/trace/SQL; IP/email/member/profile/course activity; unbounded record values; and raw lock/fence/job tokens.
Configuration contract
This issue updates only an existing manifest-approved source. It does not create, delete, transfer, or discover sources. The final #275/rollout policy must classify every configuration attribute into exactly one of:
- editable under
content.source.configure— the frozen allowlisted subset, expected to cover enabled state and only explicitly approved repository/owner/branch/ref/adapter/mount/resource/actor/credential-policy identifiers; - visible but immutable — source UUID/stable key, ownership partitions, contract versions, and other code/manifest authority;
- masked/boolean-only — whether required webhook secret and read credential bindings are configured/healthy, never either reference or value; or
- not exposed — provider payload, token, secret, raw environment/configuration, internal job/lock values, and any unapproved field.
No mass assignment is permitted. The configuration command validates the entire resulting policy, exact object permission, current source revision, global source/ownership/path uniqueness, and the accepted #275 policy digest before commit. It records a redacted history/audit entry and advances revision atomically. Network observation, checkout, parsing, and sync occur only through a separately durable after-commit request under the exact final policy; the final grooming pass must state whether a successful configuration change merely invalidates current complete evidence or also creates one confirmed follow-up operation. Until #275 freezes that behavior, engineering is blocked.
Manual sync and retry semantics
Manual sync
A manual request means “observe and, if needed, sync this source's currently configured authorized branch head.” It cannot provide a repository, branch/ref, commit/SHA, adapter, mount, credential, source policy widening, or force flag. The shared #275 command revalidates source enabled state, revision, caller/object capability, final high-risk evidence, request idempotency, current locks/pending state, and immutable policy identity, then creates/reuses the accepted run and durable intent after commit.
Exact replay is a no-op returning the original operation/run. Same key with different source/revision/reason/confirmation/policy intent conflicts. Concurrent requests converge under #275's source/run/job fences; they do not bypass coalescing or create competing checkout work.
Safe retry
Retry is available only for the accepted #274/#275 retryable terminal partial/failed classification. It creates a new linked run; it never reopens or mutates the terminal log. The service revalidates source revision, current configured repository/ref policy, immutable input/plan identity, reachability/current-head rule, retry limit, current lock/pending work, authorization, high-risk evidence, and idempotency. If the old input is no longer eligible/current/authorized, the response gives a bounded actionable reason and directs the operator to request a normal current-head sync; it never silently rolls backward or substitutes another commit.
Neither action activates a release, swaps a pointer, restores a staged release, publishes a draft by command, rolls content back, calls GitHub in the request transaction, or bypasses #274's truthful partial/failed behavior.
Confirmation, revision, idempotency, audit, and failure behavior
- Read-only list/detail/history/diagnostics are side-effect free and never refresh, reconcile, enqueue, or mutate status.
- Configuration, manual sync, and retry use the final #32/#33 production high-risk registry. Unknown/missing policy fails closed. They must show/bind exact action, source label/UUID, expected revision, scope, impact, and bounded reason; use the owner-approved explicit “are you sure” confirmation and reauthentication/API equivalent; and do not use a generic click as confirmation. No dual approval is introduced.
- Configuration requires expected source revision; command actions bind the same current revision even when they do not mutate configuration. Stale requests have no side effects.
- Idempotency is adapter-neutral and separate from webhook delivery, source/run, and durable-job fences. Rollback of the outer transaction leaves no orphan audit/operation/job; wakeup failure leaves durable work recoverable.
- Audits are append-only and use exact final actions for configuration allowed/denied/conflict, manual-sync requested/replayed/denied/conflict, retry requested/replayed/ineligible/denied/conflict, and high-risk confirmation/reauth failure where policy requires. Store actor/principal class, action, target UUID/type/label snapshot, outcome, source revision, idempotency/request/correlation digests, bounded reason/status/count/impact, changed configuration field names, and linked operation/run UUIDs only.
- Audit changes never include old/new repository/ref/path/actor/source text, secret/credential reference or value, commit/tree where restricted, edit URL, provider data, raw reason text, or configuration payload. Denied high-risk attempts follow #32/#33's no-existence-leak and redaction policy.
- Provider/configuration/queue/worker failure leaves current public authority unchanged, keeps truthful non-complete status, and presents one safe actionable code. The management surface never falls back to an alternate source, public archive, anonymous credential, staged activation, direct model edit, or synchronous provider call.
Edit-on-GitHub contract
Where content.source.edit_link.view and the object/field policy permit it, source/run/document diagnostic views may present an external GitHub link derived only from the accepted configured immutable repository/ref and normalized allowlisted source-relative path/provenance. The link cannot be supplied by diagnostic text or request input, never contains credentials/query tokens, uses safe external-link attributes, and does not imply that the website can commit, branch, open a pull request, or observe an edit synchronously.
No edit link is rendered for missing/ambiguous/cross-source/out-of-policy provenance, unauthorized users, or masked auditor views. Link denial does not hide the remaining safe diagnostic result.
Browser and accessibility scenarios
The independent tester captures and inspects desktop (~1440×900) and mobile (~390×844) screenshots under .tmp/screenshots/issue-277/ for:
/studio/content/sources/with multiple fresh/in-progress/partial/failed/stale sources and the empty/no-filter-results states;- one source detail with healthy evidence, and separate stale/partial/failed/unavailable states that never present an old success as current health;
- run history and one diagnostic detail with bounded safe codes/counts and no raw/provider/source/secret data;
- authorized configuration view plus validation error, stale-revision conflict, explicit-confirmation cancel, and accepted result;
- manual-sync confirmation plus accepted/replayed/coalesced/denied or stale result;
- eligible retry plus ineligible-current-head/revision result, showing that the old run remains immutable and a new linked run is created only on acceptance;
- content-operator, site-admin, auditor, unrelated-role, signed-out, expired/offboarded, and object-denied navigation/action visibility; and
- a permitted edit-on-GitHub link and the absent/masked state.
Screenshots must show the expected Studio page—not a debug/error/login leak—without horizontal overflow, clipped controls, credential/source-content leakage, or inaccessible status conveyed by color alone. Keyboard order, visible focus, labels/instructions/errors, confirmation semantics, aria-live status updates, table/card responsive behavior, headings, landmarks, reduced motion, and browser back-cache denial meet WCAG 2.2 AA. Test JavaScript and no-JavaScript form paths where behavior differs.
Acceptance criteria after blockers are resolved
- #32 and #33 are accepted and the issue binds every capability/action to their exact production role, object/field, reauthentication/API-proof, confirmation, revision, idempotency, audit, denial, operation, and OpenAPI symbols.
- #273, #274, and #275 are accepted and the issue replaces placeholders with exact model/DTO/query/command/recovery/operation/job/freshness/diagnostic/configuration/source-policy names, fields, enums, reason codes, and response/error schemas.
- All eight Studio and eight admin API operations above are declared in the management registry with exact permission, shared service, method, operation ID, schema, concurrency/idempotency, rate/cost, audit, confirmation/reauthentication, and test factory metadata; bidirectional registry/route/service/result/OpenAPI parity passes.
- Positive/negative role, principal, object, and field tests prove the exact capability matrix, authorized-queryset-before-lookup behavior, immediate offboarding/revocation, masked auditor projection, edit-link policy, and no object-existence leak.
- Source list/detail/history/diagnostics prove bounded pagination/filter/sort, stable ordering, truthful status/freshness mapping, immutable history, empty/no-results states, and no provider/source/secret/PII leakage. Safe GETs cause no query, status, audit, job, scheduler, network, or content mutation.
- Configuration permits only the final allowlisted fields, rejects mass assignment and policy/ownership conflicts, enforces revision plus final high-risk controls, atomically writes source/configuration-history/audit/operation evidence, and follows the accepted after-commit invalidation/follow-up rule without secret/reference exposure.
- Manual sync observes only the configured current authorized head, and retry only creates a new linked run for one eligible exact failed/partial input. Neither accepts arbitrary source parameters/SHA/force/rollback, contacts a provider in-request, reopens a log, or bypasses the four #275 idempotency/fence layers.
- Exact replay, changed-intent conflict, concurrent duplicate, stale revision, disabled source, active lock, coalesced request, pending follow-up, ineligible retry, newer/out-of-order/non-fast-forward head, outer rollback, wakeup failure, provider outage, and worker failure all return the accepted safe result with no duplicate run/job/audit or false complete status.
- Append-only audit/configuration history proves allowed, denied, stale, conflict, replay, ineligible, cancelled, queued, and failed outcomes while redaction canaries find none of the forbidden values in database rows, logs, metrics, errors, OpenAPI examples, pages, screenshots, or artifacts.
- All management responses are private/no-store/noindex with zero-TTL edge behavior; CSRF, Bearer-only API auth, CORS denial, method/media/body limits, safe errors/request IDs, rate/cost limits, and signed-out/back-navigation cache isolation pass.
- No direct-sync Studio/API route, navigation, registry entry, service, schema, or OpenAPI operation exposes candidate preview, prepare, ready, activate, rollback, force-SHA, arbitrary repository/branch/path, source creation/deletion, raw logs/payloads, credential values, or provider mutation. Existing staged compatibility is untouched unless separately owned by Slice F.
- Focused management/content/content-sync/jobs/security/audit/OpenAPI tests, migration drift, database portability, quality/type, graph-selected complete Django, full Playwright, accessibility, container, and versioned verification-plan checks pass. This rendered cross-surface issue cannot use a backend-only browser disposition.
- The independent tester recomputes the frozen plan, validates all evidence, and captures/reads every required desktop/mobile/state screenshot above; PM then accepts navigation, copy, status truthfulness, safe denials, empty/error states, responsive behavior, and consistency with #38.
- Engineer/tester/PM/on-call reports record exact base/head, graph/plan/report/artifact digests, routes/capabilities/OpenAPI operations, commands/counts, screenshot paths, and every rerun/reuse/skip/not-applicable disposition.
Explicit non-goals
- No implementation while
needs groomingremains and no guessed predecessor symbol, field, reason code, rate, proof, secret/credential, or provider/source policy. - No public reader, projection/search/graph/sitemap/feed/SEO/cache/asset cutover; no request-time provider work or change to public last-known-safe behavior.
- No webhook/HMAC/delivery fence, checkout/provider adapter, runner/upsert, source lock, durable job, reconciler/watchdog/freshness engine, schema/migration, or direct-row business invariant owned by #273–#275.
- No staged
ContentReleasecandidate/prepare/ready/activate/rollback action, pointer swap, old-SHA force sync, automatic rollback, historical-row mutation/removal, or staged contract cleanup. - No source creation/deletion/transfer/discovery, arbitrary repository/branch/ref/path/SHA/URL/host/adapter/actor/credential, cross-source adoption, hard delete, fuzzy identity join, or direct database/admin mutation.
- No GitHub commit/branch/pull-request/comment/status mutation, source script execution, local renderer/provider fallback, production credential inspection, provider/AWS/production-data operation, deploy, Actions rerun, commit, push, or merge.
- No replacement of Django admin; it remains the separately protected break-glass surface governed by #32, not the normal content-management interface.
Dependencies and final grooming exit
Blocking implementation order is accepted #32/#33, then accepted #273 → #274 → #275 with the exact first-source/rollout/configuration policy. #253/#219 and owner decisions are inherited where those predecessors require them. Public cutover (#276) is not required to build truthful read-only management status, but no end-to-end live-authority claim may be made before the applicable cutover unit passes.
After those dependencies close, PM must:
- rebase this contract against current main and the final management registry/OpenAPI conventions;
- insert exact accepted Python/service/DTO/model/job/policy/reason/schema symbols and the configuration field/effect matrix;
- verify every route and capability has one service and parity row with no obsolete activation action;
- confirm the screenshots/states remain exhaustive for the implemented surface; and
- only then remove
needs groomingand post## Grooming Complete.
Until then #277 is a precise blocked product contract, not an engineering handoff.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with _docs/PROCESS.md and the dependencies in #32, #33, and #273–#275; this issue explicitly says not to implement until their interfaces and policies are accepted. Then review _docs/specs/06-studio-and-admin-api.md and the listed architecture and security documents. Done means PM replaces predecessor placeholders, re-audits routes and fields against main, and posts “## Grooming Complete”.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- django, github, openapi, python
- Domain
- api, backend, security, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 10/100