Automate private Mailchimp account exports and attach them to recurring work

Automate private Mailchimp account exports and attach them to recurring work

Status: blocked — source is accepted/deployed; waits for #166 steady state, external credential readiness, and one HUMAN real-provider acceptance run
Tags: enhancement, portal, frontend, backend, infra, data, P1
Depends on: #166 Phase D, canonical first-write evidence, and accepted cleanup; external Dapier credential owner
Satisfied source dependency: #120 is closed completed
Not a dependency: #187 changes one-off import-test selection only and preserves Mailchimp product tests
Blocks: None
Next owner: Dapier credential owner may record sanitized readiness; after #166 cleanup, an authorized DataOps mailing-export operator performs the HUMAN checklist
Resume condition: final canonical Tasks and ordinary deployment are restored, the Dapier item exists, and the non-secret mailing configuration targets the intended post-cutover recurring Task

Final source disposition

The agent-verifiable Mailchimp export product is complete.

Accepted commit e527164429e60c573cd14162a3e9020bd3184d16 is in current source and was deployed. Tester and PM accepted the provider-neutral request/poll/download flow, durable idempotency, private storage, Artifact/Task linkage, controlled download, portal behavior, safe failures, and privacy boundary. On-Call verified deployment and unauthenticated denial without calling Mailchimp.

#120 is now closed as the completed credential-source boundary. Its deployed reader uses the Dapier-owned credential_id: "mailchimp" record through exact-key, read-only IAM and no legacy Mailchimp Secrets Manager permission.

There is no source implementation, test, IAM, bucket, route, portal, or deployment work remaining under #108 unless the bounded HUMAN run reveals a new defect.

Product classification

This is a permanent source export product, not migration/import machinery.

It requests Mailchimp’s asynchronous account-wide audiences export, persists safe run state, stores the ZIP in retained private DataOps storage, creates one private Artifact, and attaches it to recurring work. That is retained under the user decision export from the data source — keep.

#174 removed migration-only APIs, planners, checkpoints, rollback/orphan frameworks, and ongoing importer tests. It explicitly retained live exports.

#187 keeps one-off dry-run import verification out of normal CI. It explicitly leaves Mailchimp, scheduled export, and archive tests in the ordinary backend suite because those tests protect shipped runtime behavior. #108 does not depend on #187 publication, and no one-off import/export/restore command is part of this HUMAN acceptance.

Do not delete, demote, or move the mailing-export product tests to one-off coverage. Do not add migration API support, generic credential access, or compatibility fallback.

Completed acceptance

• Provider-neutral request, status polling, and download are separate from scheduling, persistence, storage, Task attachment, and presentation.
• Mailchimp uses account exports with the audiences stage and honest account-wide semantics.
• Runs have deterministic idempotency, durable pending/completed/failed state, safe retry actions, and no persisted provider download URL.
• Repeated/concurrent delivery of one run key cannot create duplicate logical provider requests, objects, Artifacts, or Task references.
• Unfinished runs are advanced under the same identity and the one-export/24-hour restriction is not bypassed.
• Completed ZIPs are checksummed and stored in retained, private, versioned, encrypted, TLS-only DataOps storage.
• Authenticated operators can list, start/advance, and download; unauthenticated calls are denied.
• Portal and API expose safe status/actions without credentials, contacts, provider URLs, or raw storage identity.
• Failure categories and logs are actionable and sanitized.
• #120’s Dapier credential reader and exact-key read-only IAM are deployed and production-verified.
• Agent source, test, PM, deployment, and public authorization gates passed.
• [HUMAN] External owner confirms the Dapier mailchimp item is populated without revealing its value or storage identity.
• [HUMAN] Post-#166 configuration references credentialId: "mailchimp" and the intended final canonical recurring Task.
• [HUMAN] One bounded real export passes the provider, private-storage, canonical Task, controlled-download, privacy/log, and idempotency checks below.
• [HUMAN] Only after that pass, the authorized owner explicitly disposes the old Zapier/Google Drive flow.

Preconditions and sequencing

External credential readiness

The Dapier credential owner may populate or re-save the item and record only:

• readiness: populated;
• provider classification: Mailchimp;
• authorization for one DataOps acceptance run.

Do not record the key, server suffix, account identity, table name/ARN, item payload, update timestamp, or Dapier/private admin evidence. This readiness action does not authorize DataOps to call the provider.

#166 steady state

Do not run the real export during repaired A/B/C, writer quiescence, Phase D execution, or cleanup.

The export may write a run, Artifact, and canonical Task reference. Wait until:

• Phase D has deployed the final steady-state application;
• canonical first-write evidence has passed;
• accepted cleanup removed transitional guards and restored ordinary push deployment;
• runtime seeds/schedules are restored;
• the intended recurring Task exists in the final table and its current identity is known privately.

#108 source is already an ancestor of the #166 chain and must not add to or alter A/B/C/D.

Final non-secret configuration

After #166 cleanup, an authorized configuration owner confirms one enabled Mailchimp configuration:

• uses credentialId: "mailchimp";
• describes account-wide audiences without contact/account-sensitive labels;
• points to the intended final recurring Task;
• contains no key, secret reference, provider download URL, raw storage URI, or legacy Task ID.

If configuration must change, use the ordinary reviewed main-push OIDC deployment path after #166 cleanup. Do not manually deploy or patch Lambda.

HUMAN acceptance run

1. Start and advance one run

Given the preconditions above and the legacy backup still active
When an authorized operator starts the current eligible export once
Then DataOps records requested or pending without exposing credential/provider details and disables duplicate activation while the request is in flight

If Mailchimp is still building the archive, use the scheduled invocation or the visible Advance / retry action on the same durable run. Do not create a different key, repeatedly poll with fixed sleeps, or bypass the provider’s one-export/24-hour limit.

2. Verify completion and canonical relationship

When the provider reports completion
Then the run reaches completed with safe filename/type/nonzero-size/checksum metadata, exactly one private Artifact is created, and exactly one reference appears on the intended final canonical Task

Re-advance the same completed run once. It must return the existing result without another logical provider request, object, Artifact, or Task reference. A stale/missing Task is a failed acceptance result: keep the archive, correct configuration through the ordinary path, and retry attachment without requesting another provider export.

3. Verify controlled access

From an authenticated portal session, use the controlled download action and confirm the private ZIP is obtainable only through the authorized short-lived response. Validate the archive only in an approved private environment.

Confirm the previously issued access expires according to the advertised server policy and requires a fresh authenticated action. Verify unauthenticated list, run, and download calls are denied without revealing whether an archive/account exists.

Never place the ZIP, extracted contacts, screenshot, provider response, signed URL, or storage location in this public repository, issue, CI artifact, shared .tmp/, or unmanaged drive.

4. Verify logs and privacy

Review the bounded run logs privately. They may contain safe transition/status categories only. They must not contain:

• API key, authorization header, or Dapier payload/identifier;
• contacts, audience members, account identity, or archive content;
• Mailchimp signed download URL;
• raw bucket/object identity or browser-visible s3:// URI;
• private Task, Artifact, run, or configuration identifiers in public evidence.

5. Decide the legacy flow

Only after all checks pass may the authorized owner disable the old Zapier → Google Drive backup and revoke its obsolete access. Do not delete historical backups in this issue.

If any check fails:

• keep the legacy flow active;
• disable the new mailing configuration only through the ordinary authorized deployment path if required for safety;
• preserve the private archive/run for diagnosis;
• rotate/revoke a credential only if the credential owner determines exposure;
• file a separate needs grooming product/security bug with sanitized failure stage and no private identifiers.

Public-safe evidence

A passing comment records only:

• UTC verification window;
• that the item/configuration were ready, without their identifiers;
• final safe status (completed);
• whether one logical request/object/Artifact/Task reference remained after replay;
• whether controlled download and expiry/reauthorization passed;
• whether unauthenticated denial passed;
• whether private log review found no sensitive data;
• whether the legacy flow was retained or disabled by its authorized owner.

Do not post exact run/configuration/Artifact/Task IDs, account labels, credential/table/bucket/object identifiers, contacts, archive metadata that identifies the account, signed URLs, headers, screenshots with private data, or provider error payloads.

Closure rule

When all HUMAN checks pass, PM may close #108 without a repository commit, provider rerun, migration/import/export test command, or application deployment.

A provider/product failure reopens implementation only through a new bounded issue. Credential absence or denied authorization remains an external blocker, not a reason for DataOps to add a fallback credential source or migration framework.

Evidence

• Tester final PASS: https://github.com/DataTalksClub/dataops/issues/108#issuecomment-4965839157
• PM final acceptance: https://github.com/DataTalksClub/dataops/issues/108#issuecomment-4965851256
• Accepted commit handoff: https://github.com/DataTalksClub/dataops/issues/108#issuecomment-4965887315
• Deployment evidence: https://github.com/DataTalksClub/dataops/issues/108#issuecomment-4970333572
• Credential-source deployment/IAM evidence: https://github.com/DataTalksClub/dataops/issues/120#issuecomment-4970815888
• Closed credential-source disposition: #120
• Migration-framework boundary: #174
• One-off test-selection boundary: #187

Out of scope

• Any agent/provider/AWS/credential/data/export/import/restore action from this reconciliation.
• A second real Mailchimp run solely to prove #120.
• Importing/restoring contacts, contact sync, campaign sending, newsletter scheduling, analytics, or a second provider.
• Generic credential APIs, Secrets Manager fallback, migration routes, compatibility paths, or importer tests.
• Deleting historical backups or disabling/revoking the legacy flow before acceptance.
• Changing #166 rollout artifacts or running during transition.

No repository file or test remains to be changed: the source implementation and deployment are already accepted. First read the preconditions and HUMAN acceptance checklist, then wait for #166 steady state and external credential readiness; done means one authorized export passes the private storage, canonical Task attachment, controlled download, privacy, and idempotency checks.