Review and optionally run the retained one-off bookkeeping PDF import

Review and optionally run the retained one-off bookkeeping PDF import

Status: blocked — Stage A PM ACCEPTED and committed at 4309a3c7395d04a016aad3c80c5cf00f69b01582 with Refs #117; exact checkpoint held off main, unpushed, and unshipped; #166/#182/#187 are satisfied; #146 and HUMAN/private stages remain pending
Tags: enhancement, human, backend, data, testing, P1
Depends on: #174/#166/#182/#187 satisfied; #146 must complete exact sandbox-provenance verification before any private source review/dry-run/write
Blocks: none; #71 is blocked only if its HUMAN owner explicitly adds this historical bookkeeping-document load to V1 sandbox acceptance
Next owner: Orchestrator after #146 completes, to integrate and push the exact accepted Stage A commit through the normal lifecycle; On-Call then verifies CI/CD; HUMAN product/data owner later decides include or defer
Resume condition for publication/private-data work: #146 completes and proves the exact target is sandbox; then the exact accepted Stage A commit reaches origin/main through normal integration/push and receives On-Call PASS; then the HUMAN owner records that #117 is included plus names the private source/review owners. Until all applicable gates pass, perform no archive/manifest read, dry-run, API/AWS call, import, export, or restore.

Product decision

Retain the raw 2024–2026 bookkeeping PDF archive importer so the data can be loaded after the destination model and sandbox are stable. Treat it as manual one-off source tooling, not as a supported migration product.

The permanent product boundary is the canonical bookkeeping model accepted under #107 and retained after #174:

• authenticated Bookkeeping accounts, transactions, active PDF documents, and many-to-many document/transaction links;
• ordinary operator document prepare/upload/complete/cancel and link APIs, which are also used by the frontend and remain product behavior;
• SHA-256 content identity and idempotent product API semantics that prevent duplicate active documents/links.

The one-off boundary is:

• backend/scripts/import-bookkeeping-documents.ts plus its archive/manifest/schema helpers;
• the manual bookkeeping:documents:import package command;
• private source archives, mapping manifest, and local operator evidence.

No Task/Card is created or changed by this import. #179/#168 do not define its data shape; #166 matters only because private financial writes must not overlap the controlled application/storage rollout or use a transitional deployment.

Corrected historical evidence

The original #117 implementation was accepted and deployed at e116bf7 / merge 5309f6c, but its migration-product acceptance is no longer the current contract.

#174 deliberately removed the parts the user rejected:

• bookkeeping ingestion/migration-only API behavior;
• migration run records, run ownership, resume/checkpoint service, reconcile, and rollback routes;
• migration-specific cleanup guidance and upload run IDs;
• the large bookkeeping importer suite and ongoing importer verification.

It retained and simplified the raw archive importer in backend/scripts/ and kept the ordinary product document/link APIs. Current source uses a straight-line flow: validate a private manifest and bounded ZIPs offline; on explicit write, authenticate through the normal operator API; prepare/upload/complete documents; create idempotent links; return sanitized counts.

Therefore the old checked criteria for persistent checkpoints, migration run IDs, exact migration reconciliation endpoints, rollback APIs, and mandatory no-op reruns are historical—not current requirements and not to be rebuilt.

#187 separately removes the retained portable dry-run-import test from ordinary backend CI. Its principle also applies here: no bookkeeping importer suite is restored, and no normal CI/deploy/schedule/runtime path executes this one-off tool. Permanent bookkeeping API tests remain because the browser and ordinary operator uploads use those APIs.

No-action and privacy boundary

This grooming authorizes no repository edit, test, AWS/provider/credential access, archive/manifest read, private data inspection, API call, import, export, restore, upload, link creation, or data mutation.

Raw archives, PDFs, filenames, manifest rows, hashes tied to private files, paths, transaction/account details, signed URLs, session material, and value-level evidence remain in an approved private/ignored workspace. Public issue/CI evidence is limited to sanitized counts, reason categories, public code SHA, target environment name, and PASS/FAIL.

Do not copy private finance material to dataops-knowledge merely because it is private operational knowledge. Raw financial evidence belongs in its approved source/runtime evidence boundary; only public-safe operational guidance belongs in the knowledge repository.

Stage A — script-only readiness (locally eligible, no data execution)

Current retained source needs a bounded safety pass before the HUMAN operation. Own only the importer and its script-local helpers/schema/package description as required. Do not change the permanent bookkeeping model or add an API.

• Remove the implicit production origin. Offline validation needs no API origin; a write requires an explicit HTTPS --api-base-url, explicit --confirm-origin, and exact target-environment confirmation. This issue authorizes sandbox only and refuses production.
• Preserve no-write as the default. The no-write path reads only explicitly supplied local archives/manifest, creates no API/client/session, and prints only aggregate counts/reason codes.
• Before the first mutating request in write mode, authenticate the current enabled operator and resolve the complete manifest target set through ordinary read APIs: every explicit transaction ID/source key is unique and present, every statement account is present/active with the intended kind, every document mapping is canonical, and all cardinality/resource limits pass. Any unresolved/ambiguous/disabled target fails with zero writes.
• Keep exact manifest/archive fingerprint validation and bounded ZIP handling from the retained source. No automatic OCR, classification, account inference, transaction matching, recursive archive import, or live provider fetch is added.
• Use only ordinary prepare/upload/complete/cancel/link APIs. The importer has no DynamoDB/S3 client, table/bucket credential, migration route, migration record, checkpoint/resume store, reconcile/orphan service, rollback endpoint, compatibility path, or API-only source lookup added for the import.
• Preserve deterministic document idempotency and permanent link uniqueness. A same-input rerun is permitted only after an interrupted/ambiguous one-shot operation and explicit HUMAN decision; do not require a second write as acceptance and do not claim a general rerunnable migration service.
• On failure, stop and emit a sanitized reason. Do not automatically delete rows, improvise rollback, continue past a changed source/target, or conceal a partial result. Any corrective data action is separately reviewed against ordinary product behavior or a separately authorized recoverability decision.
• Return a sanitized final summary for planned/created/existing documents and links plus exclusions. Never print raw names, paths, content, account/transaction identifiers, signed URLs, session values, or mapping rows.
• Keep the importer absent from Lambda/SAM artifacts, handler/router imports, deploy workflows, seeds, schedules, normal CI, and normal backend test selection.
• Do not add or restore importer unit/integration/fixture tests. Verification is source review, TypeScript script compilation, runtime-boundary inspection, and diff hygiene only; never execute the importer or one-off data commands during implementation/Tester/PM.

Stage A may receive independent Tester and PM acceptance and an explicit-path Refs #117 checkpoint commit, held off main. Integration/push waits for the post-#166/#146/#187 steady-state branch so the controlled rollout is not skipped.

Stage B — HUMAN include/defer and private source approval

After the deployment resume condition:

• [HUMAN] Decide included or deferred for the sandbox. Deferral performs no source read/dry-run/write, records the user impact, and does not block #71 unless #71 explicitly made this load launch-critical.
• [HUMAN] If included, name the archive custodian, manifest reviewer, bookkeeping data-model owner, sandbox owner, credentialed operator, private evidence/retention owner, and incident/escalation owner.
• Privately freeze the exact 2024–2026 archive set, source fingerprints, manifest, exclusions, target bookkeeping accounts/transactions, and code SHA. Historical aggregate comments are context only; they do not pre-approve current bytes or mappings.
• Every accepted PDF has one explicit document type and either explicit transaction links or an explicit unlinked approval. Statements have an existing approved account/month and explicit statement-coverage; no relationship is inferred by filename, text, or month.
• Every archive member has an approved disposition. Unsafe archive structure/resource failures cannot be waived by an exclusion. Unknown, duplicate-conflicting, sensitive, or unresolved mapping count is zero.
• Retention/disposal for raw archives, rejected items, manifest, and local run evidence is decided privately. Public comments contain aggregate status only.

Stage C — one offline no-write validation

Use the exact accepted Stage A SHA and frozen private inputs.

• [HUMAN] Run the importer once without the write flag. Confirm it performs no network/client/session/API/AWS action and reports the exact sanitized archive/occurrence/unique-document/duplicate/link/exclusion totals.
• Privately review every exclusion and mapping, bounded archive/member/byte/compression/cardinality result, and source fingerprint. Any mismatch or warning not explicitly accepted stops the operation.
• Separately confirm through ordinary authenticated read-only product surfaces that every intended account/transaction target exists in the final sandbox and that no #166 deploy/update or maintenance conflict is active. Do not add a target migration-preview API.
• Post only sanitized totals, accepted code SHA, target sandbox, and PASS/FAIL. No raw output or private digest-to-path mapping is public.

Stage D — explicit one-shot sandbox write

A write is optional and separately authorized after Stage C.

• [HUMAN] Reconfirm unchanged source/manifest/code SHA, final sandbox origin/provenance, operator identity/authorization, expected counts, bounded time/cost, and a quiet maintenance window.
• [HUMAN] Confirm existing DynamoDB point-in-time recovery and S3 versioning/retention are healthy for the permanent bookkeeping resources. This is a readiness check, not permission to export/restore or add migration rollback machinery.
• [HUMAN] Approve the exact single command with write flag, explicit API origin, matching origin confirmation, and sandbox confirmation. Available credentials do not imply approval.
• The importer completes its whole target preflight before the first prepare/upload/link mutation, then performs the accepted plan once through ordinary APIs.
• If a request fails or its outcome is ambiguous, stop. The operator records sanitized partial counts and decides privately whether the exact deterministic command may resume missing work; no blind retry or cleanup occurs.

Stage E — ordinary product acceptance and closure

• Read-only product/API evidence confirms every accepted unique PDF maps to one active canonical document, requested document/transaction/coverage tuples exist once, and created/existing/excluded totals match the approved plan. Use ordinary list/detail/link behavior, not a migration reconcile route.
• The operations manager privately spot-checks representative invoices, receipts, both business-account statement coverage, any approved private statement, linked and explicitly unlinked documents, and duplicate reuse in the authenticated Bookkeeping surface.
• No source value, filename, private URL, signed URL, account/transaction identifier, or credential appears in public logs/issues/CI; sanitized final counts and outcome are posted.
• If #71 explicitly included the load, cross-post the sanitized outcome there. Otherwise #71 remains independent.
• PM accepts the one-shot outcome and closes #117. The raw tool remains manual/outside runtime and ongoing CI until a later explicit retain/delete decision; no permanent migration machinery survives.

Acceptance criteria

• #107's permanent authenticated bookkeeping document/link product is shipped; #174 removed the rejected migration framework while retaining the raw one-off importer.
• Migration-run/reconcile/rollback/resume APIs, compatibility, and ongoing bookkeeping importer suites are absent and are not closure requirements.
• Stage A makes the retained tool explicit-target, sandbox-only, fail-closed before first mutation, sanitized, and manual-only without adding tests or migration APIs.
• #187 is shipped and ordinary CI/deploy cannot discover or execute one-off import verification.
• #166/#182 cleanup and one ordinary deploy complete.
• #146 proves exact sandbox provenance before private source or target work.
• [HUMAN] Stage B records included or deferred and, if included, accepts an exact private source/mapping with zero unresolved.
• [HUMAN] Stage C passes one offline no-write validation with no network/data mutation.
• [HUMAN] If included, Stage D authorizes and performs at most one bounded sandbox write, or an explicitly accepted same-input continuation after interruption; no production target or blind rerun.
• Stage E proves the result through ordinary permanent product reads/UI with sanitized evidence, then PM closes #117.

Test scenarios

Stage A is reviewed without private data

Given the retained one-off source tool
When engineering/review occurs
Then only source/static/TypeScript/runtime-boundary evidence is used; no importer, archive, manifest, API, AWS, import/export/restore, or one-off test command runs.

Historical migration route is requested

Given #174 removed migration support
When implementation considers run records, reconcile, rollback, orphan, resume/checkpoint, or a compatibility API
Then reject it; use the ordinary document/link product API or stop and separately groom a true product need.

HUMAN defers the load

Given the stable sandbox is available
When the owner marks #117 deferred
Then no source is opened and no dry-run/write occurs; #71 remains unblocked unless it explicitly included this data.

Offline validation finds an unsafe member or mapping gap

Given frozen private inputs
When bounded validation detects archive, fingerprint, manifest, account/transaction, or disposition failure
Then stop before network/write and resolve it privately; an exclusion cannot hide structural danger.

Write preflight finds target drift

Given an accepted offline plan
When the exact sandbox target no longer resolves every account/transaction or a deploy is active
Then the write command exits before its first mutation and requires renewed HUMAN review.

One request has an ambiguous result

Given an authorized one-shot write
When a network/API outcome is unknown
Then stop; rely on permanent content/link idempotency only after a HUMAN authorizes the unchanged deterministic continuation. Do not add a migration reconciler or blindly replay.

Out of scope

OCR/PDF extraction, automatic classification/matching, ledger transaction creation, Task/Card changes, live source/provider fetching, public finance artifacts, production data, permanent migration APIs/state/checkpoints/reconciliation/rollback/orphan management, compatibility/dual paths, importer CI/tests, a second mandatory write, manual AWS mutation, export/restore execution, edits to source repositories, and changes to #166/#146/#187/#71 beyond sanitized dependency evidence.

Start with backend/scripts/import-bookkeeping-documents.ts and its archive, manifest, and schema helpers, then inspect the bookkeeping:documents:import package command. Verify the stated runtime boundaries and TypeScript compilation without executing the importer or reading private inputs. Done requires the listed Stage A review and later HUMAN, sandbox, validation, and acceptance gates; the payload authorizes no immediate data operation.