Move Podcast Assistant operational knowledge behind the private boundary

Move Podcast Assistant operational knowledge behind the private boundary

Status: blocked — PM/Security rescope required after the Stage A metadata gate failed; three-role Stage A acceptance is stopped, and no further cleanup proceeds until Security/HUMAN define both exact batches and their private preservation/review path
Tags: assistant, podcast, docs, process-docs, testing, data, P0
Depends on: Security/HUMAN private definition of both exact batches and preservation/review path before Stage A restart or cleanup; aligned with #175 without duplicating its remediation; authorized Podcast content reviewers/private workspace and exact copy/deletion approvals [HUMAN] for Stages B-D
Blocks: removal of the remaining public-sensitive Podcast knowledge debt; any separately groomed live Podcast knowledge/runtime activation if #71 includes it
Does not block: #128's explicitly Podcast-excluding shared Telegram rollout, or #71 sandbox acceptance when Podcast execution is explicitly deferred
Next owner: Security with PM; HUMAN/Security owners must resolve the batch-definition and historical-preservation prerequisite before any role resumes Stage A or cleanup work
Resume condition: Security/HUMAN privately record the exact definition and preservation/review ownership for (1) the 67 currently tracked candidate paths and (2) the 51 tree-deleted but history-retained Podcast example paths, including how the latter relates to #175. PM must then accept the written rescope before Stage A restarts. Until then, do not scan more candidates, accept three-role Stage A output, alter the cleanup batches, or touch historical material.

Current evidence and corrected premise

The private destination is no longer hypothetical:

• 1b6e9b2661da87d91dc36531ca5fd8b86f0a8d7f moved the general content/ corpus—2,127 documents and 1,768 images—to private DataTalksClub/dataops-knowledge and removed the obsolete repository scaffold.
• 54e9b96 pointed the portal at that repository; 13bf00a moved content validation to a private checkout. The portal and CI boundary therefore exist independently of this issue.
• The one-session handoff was deleted by 990e046 after its migration-only guidance became stale. It is not a source of truth and must not be reconstructed or treated as migration evidence.
• The Podcast Assistant categories were deliberately left for separate review. The current public tree still contains exactly 67 tracked paths: data/ 8, knowledge_base/ 56, process/ 1, templates/ 1, and podcast_examples/ 1 under assistants/podcast/.
• Independent git ls-tree metadata at current origin/main (bfb5be058d904f7ee1038b65c456c5c1a8b94853) reproduces 67 = data 8 / knowledge_base 56 / process 1 / templates 1 / podcast_examples 1.
• Migration-era commit b321b8c7c8b7bd27ac8570e5d37eb9d105104e55 had the former 118 = 8/56/1/1/52 aggregate. The immediately preceding tree of reachable commit 9d59839aba974e98552ebcbb8fd97084a3fa5210 also had that 118 aggregate; 9d59839 changed it to 67 = 8/56/1/1/1, reducing Podcast examples by 51 while leaving the other four category counts unchanged.
• Those 51 paths were removed from the working tree but are retained by reachable public Git history. Working-tree deletion does not erase history. They are a distinct history-retained batch, not an addition to the active tracked-candidate batch. Their preservation/review and any history remediation remain owned by #175/Security; do not duplicate that response, inspect deleted blobs, restore them into #144, or fold them into the active cleanup without an explicit PM/Security amendment.
• Public source still refers to those locations from assistant build/search/import tooling and documentation, and the planning validator names the process/template paths. Removing files without updating those dependencies would leave a broken or misleading product surface.

Only aggregate category/count information is public evidence. Stage A must not open or quote candidate bodies. Individual filenames, private links, text, screenshots, generated documents, real-person context, digests, and destination paths must not be copied into this issue, public commits, CI logs, or public artifacts.

Product boundary

This issue completes the mandatory repository-boundary cleanup for Podcast operational knowledge. It is not permission to activate Podcast execution.

Keep public:

• Podcast product code, package metadata, lock files, CLI/runtime interfaces, and synthetic tests that do not embed operational knowledge;
• minimal schemas and intentionally synthetic fixtures needed to prove code behavior;
• public-safe architectural and operator-facing documentation that does not reveal private repository internals or content.

Move out of public Git after private review:

• process instructions, reusable operational templates, assistant prompts/checklists, reviewed private reference material, episode-specific examples, real-source datasets, and generated operational documents;
• unknown or ambiguous items, which are private-by-default until resolved.

Raw inbox inputs, guest-specific outputs, recordings/transcripts, run logs, drafts, and other runtime artifacts belong outside public Git and outside the knowledge source when they are artifacts rather than curated knowledge.

There is no compatibility phase: once the accepted public cleanup lands, code and tests must not fall back to the removed public paths or retain duplicate public copies.

No-action boundary

Grooming and Stage A authorize no candidate-content read, private-repository checkout, credential or provider access, copy, deletion, rewrite, commit, push, deployment, runtime/configuration change, import/export/restore, or data mutation. Do not modify the original Podcast Assistant source repository.

No public issue or CI artifact may contain the private manifest, filenames that reveal operations, content excerpts, real identities, private paths/links, credentials-adjacent instructions, or provider payloads.

Stage A — public metadata and code-dependency audit (stopped pending PM/Security rescope)

The Process Curator metadata gate FAILED. Three-role Stage A acceptance is stopped. The existing source-reference work below is non-accepting, pre-rescope input; it does not authorize additional scanning or establish an accepted unresolved count. Use Git metadata and public source only after an authorized restart; never open the 67 candidate files or the 51 history-retained example paths/blobs.

• PM reproduced the exact aggregate count 67 and category split 8/56/1/1/1 from tracked-path metadata. The changed historical delta stopped the stage instead of silently broadening the batch.
• After Security/HUMAN define both exact batches and PM accepts the rescope, restart or re-scope the remaining audit without broadening either batch.
• Defer the per-path manifest—path, extension, tracked state, byte size, last revision, and SHA-256—to the approved private workspace at the start of Stage B. Publish only category/count totals.
• Map every public code, validator, test, README, planning-doc, packaging, and runtime reference to the five candidate categories without reading the candidate content.
• Classify each reference as product code to keep, one-shot/operator tooling to keep but parameterize, public documentation to correct, test dependency to replace with synthetic input, or obsolete reference to delete. This reference classification is about source behavior, not the candidate content disposition.
• Architect defines the smallest no-fallback boundary for an absent/private Podcast knowledge source. Assistant Engineer confirms which behaviors require a knowledge input and which code can remain usable with injected synthetic fixtures.
• Process Curator posts a sanitized issue comment containing only aggregate counts, reference counts by source area, unresolved reference count, and gate result.

Stage A produces no repository change and no private copy. Acceptance of Stage A unlocks only the private HUMAN review, not deletion or activation.

Stage A implementation-ready ownership map

A PM/Security rescope is required. The currently tracked aggregate is 67 = 8/56/1/1/1 at current origin/main; the distinct history-retained delta is 51 Podcast example paths. Neither aggregate authorizes Stage A acceptance or cleanup.

A pre-gate scan found 17 literal full-prefix occurrences across 4 public files: 15 in three durable documentation files and 2 in the planning validator. Additional functional dependencies are mapped below from root-relative defaults, generated-output paths, and the process prompt; they do not contain the full public prefix. Those classifications are non-accepting input because the metadata gate failed, so no unresolved count is reported as zero. Do not continue scanning or request three-role signoff until Security/HUMAN define both batches and PM accepts the rescope.

PM rescope record

• The former aggregate-path finding was stale. Current tracked-path metadata freezes the remaining batch at 67 = 8/56/1/1/1.
• Reachable commit 9d59839aba974e98552ebcbb8fd97084a3fa5210 accounts for the isolated delta: it deleted 51 example-category paths and changed none of the other four categories.
• Prior evidence named unreachable variants of the migration commits. It is corrected to reachable 1b6e9b2661da87d91dc36531ca5fd8b86f0a8d7f, 54e9b96, 13bf00a, and 990e046.
• The pre-gate source-reference inventory found 17 literal occurrences across 4 files and the root-relative dependencies mapped below, but its unresolved count and ownership classifications are not accepted while this gate is failed. No candidate filename or body is published.

Distinct exact batch prerequisite

Security/HUMAN must privately define both exact batches before further cleanup:

1. Active tracked batch: the 67 currently tracked paths under the five category roots below.
2. History-retained batch: the 51 Podcast example paths removed by 9d59839aba974e98552ebcbb8fd97084a3fa5210 but retained in reachable public history. Define preservation, private review, incident handling, and ownership with #175/Security without enumerating or restoring them here.

Public evidence remains aggregate-only. The active cleanup batch stays at 67 unless PM and Security explicitly amend it after the prerequisite is satisfied.

Currently tracked candidate boundary

Until amended, the active candidate set is limited to tracked paths under these already-public category roots:

assistants/podcast/data/**
assistants/podcast/knowledge_base/**
assistants/podcast/process/**
assistants/podcast/templates/**
assistants/podcast/podcast_examples/**

Stage A may record filename, extension, tracked state, byte size, source revision, hash, and already-public frontmatter keys/IDs only in the approved private manifest. It must not read, quote, classify from, or publish candidate bodies. The public issue receives only aggregate category/count, source-reference counts by area, unresolved count, and gate result. Any count delta or needed path outside these roots stops work for PM/Security rescoping.

Public-source ownership for the later Stage D change

Only the following public files/file groups are currently assigned to #144. This is a source-dependency map, not authorization to edit them before Stages B/C and the deletion gate.

1. Product/runtime seams to keep and adapt

   • assistants/podcast/heru_runner.py: remove the repository-local process-file assumption. Knowledge-required behavior must receive an explicit approved input and fail closed with a safe actionable error when absent.
   • assistants/podcast/scripts/search_podcast_kb.py: require an explicit search-data/bundle input; no removed-path default, embedded corpus, private-repository checkout, network fetch, or silent fallback.

2. One-off/operator tooling to retain but isolate

   • assistants/podcast/scripts/build_podcast_knowledge_base.py
   • assistants/podcast/scripts/import_actual_podcast_questions.py

   These remain manual one-off tools, not product/runtime features. Require explicit source and output locations; refuse output into the prohibited tracked roots; do not run from normal CI, deploy, schedules, APIs, or ordinary test selection. They do not justify a migration framework, migration API, compatibility route, or retained public sample corpus.

3. Tests and synthetic boundary

   • assistants/podcast/tests/test_heru_runner.py and assistants/podcast/tests/test_search_podcast_kb.py: keep in the ordinary suite only with minimal, obviously synthetic temporary inputs and missing-input/fail-closed coverage.
   • assistants/podcast/tests/test_build_podcast_knowledge_base.py and assistants/podcast/tests/test_import_actual_podcast_questions.py: because they exercise one-off tooling, remove them from ordinary CI selection. If retained, place them behind a clearly named manual one-off boundary with synthetic temporary inputs; #144 Tester does not execute that manual command.
   • No test may retain a candidate filename, real-looking identity, source-derived prose, private path/link, provider payload, or copied operational structure.

4. Validation and CI

   • scripts/validate_planning_docs.py: remove the two required public Podcast process/template paths and enforce absence of all five prohibited tracked roots without printing discovered filenames/content.
   • .github/workflows/assistant-ci.yml: keep the public product/synthetic checks and the boundary guard; do not checkout private knowledge, request a knowledge token, or select manual one-off/provider integration tests.
   • Prefer extending the existing planning-validator test seam. A new focused public boundary test is allowed only if it is synthetic, isolated, and does not enumerate candidate filenames.

5. Durable public documentation

   • assistants/podcast/README.md
   • docs/decisions/dataops-knowledge-repository.md
   • docs/import-log.md
   • docs/operations-manager-platform-jtbd.md

   Correct obsolete public-path instructions and history while documenting only the durable public/private boundary. Do not disclose private destination internals, manifests, candidate names, or instructions that make removed paths appear usable.

Absolute excludes for #144

• Do not open or quote candidate bodies, inspect private sources, inspect deleted historical blobs, checkout deleted content, or enumerate candidate filenames in public evidence.
• Do not edit backend/**, frontend/**, infra/**, ../aws-infra/**, or any original source repository.
• Do not edit #128/#166/#71 implementation, add provider/Telegram/model/AWS behavior, provision credentials, or activate Podcast execution.
• assistants/podcast/inbox/** and assistants/podcast/documents/** are outside the frozen 67-path deletion batch. Keep runtime artifacts ignored/untracked; if tracked operational material is discovered there, stop and open a separately approved scope rather than silently broadening #144.
• assistants/podcast/tests_integration/**, package metadata, and lock files have no intended #144 change. Touch them only if the independent sanitized source-reference inventory proves a direct boundary dependency and PM explicitly amends this map.
• Do not add a compatibility layer, dual path, public fallback corpus, auto-checkout, migration API/framework, history rewrite, import/export/restore run, private-to-public rollback, or #175 remediation inside #144.

Stage A exit packet

Stage A cannot close on the current failed gate. Only after Security/HUMAN define both exact batches and PM accepts the rescope may the three assigned roles produce a new sanitized packet with: aggregate counts for each authorized batch; public source-reference counts grouped by the five ownership areas above; each reference classified to a Stage D action; unresolved reference count 0; one sanitized statement of the current default-CI selection status for one-off tests plus their assigned Stage D isolation action; and explicit pass/fail against every absolute exclude. No filenames, bodies, hashes, private locations, identities, or provider data belong in that packet.

Stage B — authorized private classification and batch approval

The destination repository already exists; this stage does not recreate it or reuse the portal token casually. HUMAN/Security owners must authorize the Podcast-specific review and least-privilege identities.

• [HUMAN] Security/HUMAN record the exact active 67-path batch and the distinct history-retained 51-example batch, including preservation/review ownership for the latter and its non-duplicative relationship to #175.
• [HUMAN] Name the private destination/content owner, authorized content reviewers, Security reviewer, retention owner, and incident/escalation owner; approve a private workspace where candidate content and manifests may be handled.
• [HUMAN] Confirm least-privilege source/destination access, branch/review rules, backup/retention, secret scanning, auditability, and how private evidence is exchanged without public exposure.
• In the private workspace, assign every one of the 67 entries exactly one disposition: private curated knowledge; private/raw source or artifact storage; approved minimal synthetic public fixture; generated/duplicate/obsolete; secret/incident quarantine; or unresolved.
• Each disposition has an owner, rationale, retention decision, and provenance. Filename alone is never sufficient; unresolved must be zero before copy or cleanup approval.
• Any credential, personal, guest, sponsor, finance, or other sensitive exposure is quarantined and follows the private incident process. Do not quote or identify it publicly.
• [HUMAN] Approve the frozen exact copy batch and any transformations/redactions. Approval does not authorize public deletion yet.

Stage C — private copy and integrity proof

• Copy before deleting. Preserve an immutable private source manifest and destination revision with sizes/hashes for every item in the frozen batch.
• Verify one-to-one coverage and byte/hash equality where no transformation is intended; transformed or curated derivatives have separately approved private lineage.
• Verify private visibility, authorized/unauthorized access, retention/backup, and rollback evidence without publishing repository internals or identities.
• Record only sanitized totals, destination revision/digest evidence, mismatch count, and gate result here. Any mismatch, access drift, or unreviewed transform aborts public cleanup.
• [HUMAN] Accept the verified private destination revision as source of truth and approve the exact frozen public cleanup batch.

Stage D — public source cleanup (implementation stage)

Only after the PM/Security batch prerequisite, Stage C, and the exact HUMAN deletion approval:

• Delete all 67 classified operational-knowledge paths from the public tree; no duplicate, archive, encoded fixture, generated copy, Git LFS pointer, or compatibility fallback remains.
• Update Podcast code, build/search/import tooling, documentation, planning validation, packaging, and other Stage A references to the accepted boundary. Kept one-shot tooling accepts an explicit authorized input/output location and is never invoked by normal CI, deploy, schedules, or product APIs.
• Replace tests with the minimum demonstrably synthetic fixtures outside the removed operational paths. Fixtures contain no real person, episode, contact, prompt, private link, provider payload, or source-derived prose.
• The public assistant fails closed with an actionable safe error when private knowledge is required but unavailable. It never guesses, silently uses embedded knowledge, or reaches a private repository from public CI/forks.
• Add a narrow guard that rejects reintroduction of the five prohibited operational path categories while allowing reviewed code and explicitly synthetic test fixtures.
• Update durable repository-boundary documentation so content/ and Podcast operational knowledge have one consistent rule. Do not restore or replace the deleted session handoff.
• Do not rewrite public Git history under this issue. Security owns any separately groomed historical-exposure response.

Stage D follows the normal Software Engineer → Tester → PM → explicit-path commit → merge/push → On-Call lifecycle. No private content or manifest is staged in the public commit.

#71 and optional runtime activation

#71 owns the HUMAN sandbox scope decision:

• If Podcast execution is deferred, #144 still completes Stages A-D because the public-knowledge boundary is mandatory. The Podcast capability remains explicitly off/unavailable; no credential, provider smoke, private runtime fetch, or activation work is required to close #144.
• If Podcast execution is included, #71 must name a separately groomed activation owner after the private source of truth is accepted. That follow-up defines a typed/versioned immutable knowledge bundle/provider, least-privilege delivery, fail-closed authorization/digest behavior, leak-safe logs/context/artifacts, rollback, and a bounded HUMAN canary.
• #144 may block that activation owner, but activation does not block #144's cleanup. Do not bundle a live provider, Telegram action, model/provider call, or production canary into the public deletion commit.
• #128 remains Podcast-excluding and is never evidence for Podcast knowledge readiness.

Acceptance criteria

• The general process corpus is private, and portal/CI consumers use the private knowledge repository; this does not claim the 67 Podcast paths moved.
• The deleted migration handoff is retired and not used as current authority.
• PM reproduced current tracked metadata as 67 with aggregate split 8/56/1/1/1; the discovered history-retained 51-path delta stopped Stage A acceptance.
• [HUMAN] Security/HUMAN privately define both exact batches—active tracked 67 and history-retained 51—including private preservation/review and the #175 ownership boundary; PM accepts the resulting rescope.
• After the authorized rescope, Stage A maps every applicable public source dependency without candidate-content inspection.
• [HUMAN] All 67 entries receive one approved private disposition with zero unresolved, and the exact copy batch is authorized.
• The private copy/destination revision passes coverage, hash/transformation, access, retention, backup, and rollback review before public deletion.
• [HUMAN] The exact public cleanup batch is approved after private integrity proof.
• Stage D removes all 67 public operational paths and every fallback/duplicate, updates all references, and retains only code plus minimal approved synthetic fixtures.
• Public checks pass without private repository credentials or content, and guards reject reintroduction without leaking names or contents.
• Tester and PM accept sanitized public evidence; On-Call confirms normal CI/CD for the cleanup SHA.
• #71 records Podcast execution as deferred or links a separate accepted activation owner; no live activation is inferred from #144.

Verification plan

Stage A reports exact commands and exit codes for metadata counts and source-reference inventory only. It does not run or inspect candidate content.

Stage D Tester verifies, from a clean public checkout with no private knowledge checkout or credentials:

• tracked-path absence and reintroduction-guard behavior for all five prohibited categories;
• focused Podcast unit/CLI behavior using only synthetic fixtures, including missing/private-source fail-closed behavior;
• planning-document validation and all source references identified in Stage A;
• package/build behavior proving no operational files are embedded;
• repository scans for prohibited raw categories, private links, real identities/context, generated operational artifacts, secrets, and destination internals;
• git diff --check and the relevant normal public CI workflow.

Private Stage B/C verification occurs only in the approved workspace. Public evidence is limited to sanitized category/disposition totals, unresolved/mismatch counts, immutable revision/digest status, reviewer gate results, and links that reveal no private location or content.

Test scenarios

Metadata total changes

Given the distinct 67 tracked and 51 history-retained findings
When cleanup scope is not defined privately for both batches
Then keep three-role Stage A stopped and require PM/Security scope resolution before changing either batch or touching history.

Candidate looks harmless

Given a filename or extension appears public-safe
When no authorized content review has approved it
Then keep it private-by-default; do not infer safety from its name.

Private copy differs

Given the approved copy batch
When any item is missing, changed unexpectedly, inaccessible, or transformed without lineage
Then abort public cleanup and preserve both private manifests for review.

Public checkout has no private access

Given an ordinary fork or CI job with no knowledge credentials
When Podcast code/tests/build run
Then synthetic behavior passes and knowledge-required behavior fails closed without attempting private access or exposing expected content.

Podcast is deferred in #71

Given Stages A-D are accepted
When #71 classifies live Podcast execution as deferred
Then #144 may close with the capability off and no provider/Telegram/model action.

Podcast is included in #71

Given the private source is accepted and public cleanup is complete
When #71 includes Podcast execution
Then a separate activation issue owns delivery/auth/canary/rollback; #144 itself performs no live effect.

Out of scope

Candidate-content inspection before Stage B authorization; private repository creation; edits to the original Podcast Assistant source repository; public Git history rewriting; credentials or provider access; Telegram/model/provider effects; production activation; AWS/IAM/deploy changes; raw data import/export/restore; runtime artifact migration; and changes to #128, #166, or #71 beyond sanitized dependency evidence.

No implementation can start until Security/HUMAN define the two batches and PM accepts the rescope; begin by reading the blocked Stage A and ownership sections. After authorization, inspect assistants/podcast/heru_runner.py and assistants/podcast/scripts/search_podcast_kb.py alongside the mapped public references. Done means the approved boundary cleanup is complete with no public fallback or duplicate operational knowledge, while preserving the stated public-safe code and tests.