Add authenticated, idempotent email-document intake into private DataOps storage

Add authenticated, idempotent email-document intake into private DataOps storage

Status: blocked — source is accepted/deployed; waits for #166 steady state and one credentialed HUMAN sandbox rollout exercise
Tags: enhancement, backend, infra, data, P0
Depends on: #166 Phase D, canonical evidence, and accepted cleanup; credentialed HUMAN sandbox operator
Satisfied source dependencies: #31 and #107 are closed
Blocks: None
Next owner: after #166 cleanup, credentialed HUMAN sandbox operator with authority over the test secret and temporary private source object
Resume condition: ordinary deployment is restored and the operator can perform one bounded accepted/duplicate/conflict/rotation/download/cleanup exercise without exposing credentials, storage identities, document metadata, or private evidence

Final source disposition

The agent-verifiable email-document intake product is complete.

Commit d28dc46 is an ancestor of current source and origin/main. Tester and PM accepted it. On-Call confirmed that it deployed after its required secret configuration was supplied and that unauthenticated requests reach the dedicated fail-closed machine-auth boundary.

Current product source remains in:

• backend/src/routes/emailDocuments.ts
• canonical Intake and Artifact repositories;
• the generic authenticated private Artifact download route;
• backend/src/router.ts;
• infra/template.full.yaml;
• the OIDC deployment workflow;
• focused endpoint and infrastructure tests;
• docs/email-document-intake.md.

There is no source, schema, route, IAM, bucket, KMS, credential-loader, documentation, or automated-test work remaining under #109 unless the HUMAN exercise reveals a new defect.

Product classification

This is a permanent live intake boundary, not migration framework residue.

POST /api/v1/intake/email-documents is the machine-to-machine entry point for future email envelopes and document descriptors. It verifies allowlisted private source objects, copies accepted bytes into retained DataOps-managed private storage, creates canonical sensitive Intake/Artifact records, and exposes documents only through authenticated controlled download.

Its conditional reservation, exact replay, conflict rejection, bounded partial recovery, and credential rotation are ordinary reliability/security behavior for an externally delivered request. They are not migration planning, checkpoints, rollback, orphan management, backfill, or compatibility.

The #174 cleanup decision therefore does not remove this route or its ordinary product tests. The retained endpoint must not gain:

• a migration-run API;
• generalized import planner/checkpoint/resume/rollback/orphan state;
• bulk historical-document loading;
• a compatibility alias or dual route;
• ongoing tests that invoke a one-off historical import.

The remaining sandbox exercise is a one-time deployment acceptance gate. It is not part of normal CI and must not be converted into a reusable migration/import/export/restore suite.

Canonical runtime contract

• The versioned M2M endpoint accepts one bounded envelope plus zero to 25 descriptors for original attachments or sender-rendered PDFs.
• JSON contains metadata and stable private S3 source references only—never attachment bytes, base64, raw email bodies, credentials, or presigned URLs.
• Dedicated rotatable header authentication runs before portal/session bypass or any write. Missing/wrong credentials are indistinguishable and non-revealing.
• Source bucket/prefix, size, type, and lowercase SHA-256 metadata are verified before copy or Artifact creation.
• Accepted documents are copied into encrypted, versioned, public-blocked, TLS-only retained DataOps storage and registered as sensitive review-needed Artifacts.
• A metadata-only TODO email may create one sensitive Intake with no Artifact.
• Normalized recipient route + Message-ID is the idempotency identity. Exact/concurrent replay reuses the same result; changed immutable content returns 409; partial retry only completes missing work.
• Responses and structured logs expose safe status/correlation categories only, with no sender, subject, Message-ID, filename, source/destination identity, checksum, credential, document bytes, signed URL, or AWS exception.
• Private document access remains authenticated and short-lived.
• The application does not receive/poll mail, render HTML to PDF, inspect document content, perform OCR/bookkeeping matching, or auto-approve.

Completed acceptance

• Metadata-only intake creates one sensitive Intake and no Artifact.
• Synthetic attachment/rendered-PDF descriptors are verified, copied, registered, and linked safely.
• Exact/concurrent replay, partial recovery, and conflicting replay have deterministic no-duplicate behavior.
• Authentication, validation, payload/object limits, allowlisting, metadata checks, safe status codes, and no-write failures are covered.
• Credential refresh/rotation logic, rate limiting, least-privilege S3/KMS/Secrets Manager, and retained private bucket controls are declared and tested.
• Stored records, responses, and logs exclude raw/signed/secret/private material.
• Public-safe contract/rotation/remediation documentation is present.
• #108 mailing exports remain independent.
• Tester PASS, PM acceptance, commit, deployment, and unauthenticated fail-closed smoke are recorded.
• [HUMAN] After #166 cleanup, one bounded temporary-object/test-credential exercise passes accepted, exact duplicate, conflict, rotated-old-credential denial, authenticated download, unauthenticated denial, private log review, and cleanup.

#166 publication sequencing

#109 source is already published and is an ancestor of the #166 rollout chain. It must not add to or alter A/B/C/D.

Do not perform the sandbox exercise during repaired A/B/C, writer quiescence, Phase D, or cleanup. Although it does not write the Tasks table, it creates retained Intake/Artifact/object/session evidence and changes credential state. Wait until:

• Phase D and canonical evidence pass;
• accepted cleanup removes transitional guards and restores ordinary main-push deployment;
• the final Lambda artifact is active;
• normal authenticated portal/private-download behavior is available;
• an authorized maintenance window prevents interference with any real upstream sender.

A pass against that steady-state artifact is final #109 rollout evidence. No source commit or application deployment is required solely to close the issue.

HUMAN sandbox exercise

This issue authorizes no agent credential access or external mutation. The following is performed only by an authorized human after the preconditions above.

1. Prepare synthetic private inputs

• Use one clearly synthetic PDF containing no real invoice, contact, email, finance, sponsor, or operational information.
• Place it under an already approved private sandbox source prefix with declared PDF type, positive size, and matching lowercase SHA-256 object metadata.
• Use a synthetic recipient route and envelope values that reveal no real address, Message-ID, sender, subject, or account identity.
• Prefer the existing managed transfer prefix. Do not add broad cross-account S3/KMS grants merely for this smoke.

2. Establish a controlled credential window

• Confirm the upstream sender is paused/inactive or the credential owner has approved a maintenance window.
• Rotate the pre-created intake secret through its authorized owner to a temporary test value without copying it into commands, logs, issue evidence, source, or GitHub text.
• Do not create a second permanent auth path, inline Lambda value, Basic/bearer fallback, or query/body credential.

3. Verify accepted and idempotent behavior

Submit one valid authenticated request:

• response is 202 accepted;
• one sensitive Intake and one sensitive review-needed Artifact exist;
• one managed private copy exists;
• no source/destination URI, signed URL, credential, document bytes, or private metadata appears in the response.

Replay the exact request:

• response is 200 duplicate;
• the same logical Intake/Artifact/copy is reused;
• no duplicate history noise or object appears.

Submit a changed immutable manifest under the same identity:

• response is 409 idempotency-conflict;
• the accepted record is unchanged;
• no second Artifact/copy is created.

Do not run bulk, historical, migration, import, export, or restore data.

4. Verify rotation denial

After the new credential is active:

• the new value authorizes without redeploy;
• the old value and missing/wrong values receive the same non-revealing 401;
• rejected requests create/copy nothing;
• no credential value or secret identifier appears in the evidence.

Do not use rapid fixed sleeps to infer cache expiry. Exercise the documented refresh-on-mismatch behavior and observe the response.

5. Verify controlled private download

From an authenticated browser session, request the accepted Artifact through the controlled-download route:

• authorized access returns a short-lived private action and the synthetic PDF is retrievable;
• API/DOM does not expose a raw s3:// URI;
• unauthenticated access returns 401/403 and discloses no Artifact/object metadata;
• after the advertised expiry, a fresh authenticated action is required.

Inspect the file only in an approved private environment. Do not attach it or a private screenshot to GitHub/CI.

6. Verify logs and clean up

Privately review accepted, duplicate, conflict, unauthorized, rotation, and download log entries. Confirm they contain only safe categories/counts/correlation hashes and none of:

• sender, recipient, subject, Message-ID, filename, checksum;
• source/destination bucket, prefix, key, URI, or signed URL;
• credential, secret identifier, token, session, raw request/document, or AWS exception.

Then:

• remove the temporary source object;
• revoke/replace the temporary credential with the approved ongoing state;
• remove any temporary external grant/configuration;
• retain only the managed synthetic Intake/Artifact/object evidence that the authorized owner explicitly approves for the sandbox record.

If cleanup cannot be proven, the gate fails.

Public-safe evidence

A passing issue comment records only:

• UTC verification window;
• final accepted/duplicate/conflict/old-credential/unauthenticated outcome codes;
• that exactly one logical Intake, Artifact, and managed copy remained;
• that authenticated download and expiry/reauthorization passed;
• that private logs contained no sensitive fields;
• that the temporary source, credential, and grants were cleaned up.

Do not post real or synthetic envelope values, exact Intake/Artifact/correlation IDs, secret/table/bucket/key/KMS identifiers, checksums, credentials, request headers/bodies, signed URLs, document contents, provider/private links, or screenshots containing private data.

Closure and failure routing

After a complete HUMAN pass, PM closes #109 without a repository commit, test rerun, AWS inspection, or deploy.

If the runtime behavior fails, leave #109 open and file a separate needs grooming security/product bug with only the sanitized stage and response class. Do not fix it by adding a migration route, alternate credential, broad IAM, source fallback, or compatibility path.

If the authorized operator or maintenance window is unavailable, keep #109 blocked with human; absence of authority is not a source defect.

Evidence

• Tester PASS: https://github.com/DataTalksClub/dataops/issues/109#issuecomment-4965370049
• PM acceptance: https://github.com/DataTalksClub/dataops/issues/109#issuecomment-4965374507
• Commit/human handoff: https://github.com/DataTalksClub/dataops/issues/109#issuecomment-4965399774
• Deployment/fail-closed smoke: https://github.com/DataTalksClub/dataops/issues/109#issuecomment-4970333763
• Migration-framework boundary: #174
• Current #166 rollout tracker: #166

Out of scope

• Receiving/polling email or changing the upstream sender.
• HTML-to-PDF rendering, OCR/classification, bookkeeping matching, or auto-approval.
• Historical invoice/archive loading or any import/migration/restore.
• A new UI, document database, compatibility route, or generic credential API.
• Mailing-list exports owned by #108.
• Any agent AWS/provider/credential/data action from this reconciliation.

Read docs/email-document-intake.md and the listed route, router, infrastructure template, and focused tests to understand the completed source. Wait for #166 cleanup and the documented maintenance window; done means an authorized human completes the bounded accepted, duplicate, conflict, rotation, download, logging, and cleanup exercise without a source change.