Roll out and cut over the Telegram conversational MVP

Roll out and cut over the Telegram conversational MVP

Status: source, reviews, OIDC policies, and normal-CI all-off deployment are complete; the remaining staged HUMAN rollout waits for #166/#182 to finish the canonical Card/Task cutover and restore a stable ordinary deployment baseline
Tags: enhancement, assistant, backend, infra, testing, human, P1
Parent: #122
Depends on: #121 and #124-#127 source/default-off prerequisites are satisfied and their remaining live evidence is owned by this rollout; #129/#130/#131 are closed/effective; live canaries wait for #166 A/B/C → #182 preflight → D → cleanup plus one successful ordinary all-off deployment
Blocks: live-acceptance closure of #121 and #124-#127, then closing #122
Next owner: #166 orchestrator/credentialed operator for the reviewed storage sequence; after cleanup and normal deployment, the authorized HUMAN Telegram/provider owner with On-Call
Resume condition: #166/#182 record accepted live sandbox outcomes, canonical writers are reopened on the final empty Tasks schema, temporary phase controls are deleted, ordinary push-triggered OIDC deployment is restored, and one normal main run succeeds with all six conversational controls off. Then execute the existing HUMAN matrix one normal OIDC stack update at a time; no manual Lambda patch or provider action outside the explicit gates.
Architecture: _docs/CONVERSATIONAL_AGENT_PLUGIN_ARCHITECTURE.md

Current review state

The repaired #128 source and E2E bootstrap passed Assistant Engineer, Architecture, Tester, and PM review and are integrated. #121 and #124-#127 remain open only because their live Telegram/provider/runtime outcomes are deliberately supplied by this consolidated rollout; they are not separate source blockers. Gate A is effective, #129 and #130 are closed, and corrected observability Gate B #131 is closed. #140/#143/#136 remediation is complete; #141 is unrelated and nonblocking.

Normal push run 31545689094 at 5ac8b39c39e358cdec41b192bad984b8a46a2daa used the exact six all-off controls, created the accepted dark realization (one rollout topic, three retained custom log groups, and three disabled generated schedule rules), deployed the conversational functions/queues/recovery resources, reached terminal UPDATE_COMPLETE, completed runtime seeding, and passed deployed smoke with no relevant authorization denial. The accepted transforms retain exactly 17 condition-gated alarms; dark mode correctly instantiates zero alarm resources. Run 31711997388 later reconfirmed the stable all-off normal OIDC deploy, seed, and smoke path.

No HUMAN webhook/poller inspection, alarm subscription confirmation, Telegram journey, z.ai/Groq/vision/Typefully call, alarm canary, flag enablement, or rollback has run. Those criteria remain open.

#166 is now the active sequencing boundary. Its repaired A/B/C chain and Phase D preparation are local/unshipped and no live phase has run. The todo canary creates a Task, so starting #128 before the approved table replacement would write into the transitional table immediately before deletion. All #128 live stages therefore wait until #166 Phase D/cleanup (including #182's final no-replacement Cards preflight) restores canonical writers and one following normal all-off deployment succeeds.

Product scope

Ship one private-Telegram conversational rollout for the two implemented
proposal capabilities:

• todo: propose, revise, explicitly approve, and create one actor-owned todo;
• Typefully: confirm typed public source, propose/revise, explicitly approve,
  and create one unscheduled/unpublished/unshared saved draft.

#126 continues to own todo/coordinator/outbox semantics. #127 continues to own
Typefully source, approval, account, request, result, and reconciliation
semantics. #128 owns only rollout integration, dark maintenance, readiness,
telemetry, deployment wiring, staged HUMAN verification, and switch rollback.

Exactly six deploy-owned controls

Expose exactly:

Telegram ingress: off | on
execution leasing: off | on
enabled plugins: none | todo | typefully | todo,typefully
Typefully external execution: off | on
voice: off | on
photo: off | on

One immutable strict snapshot accepts only canonical lowercase values and a
sorted unique plugin subset. All code, SAM, and workflow defaults are off.
Reject missing, whitespace-bearing, duplicate, unknown, retired, or
contradictory controls.

Derived behavior remains:

• runtime availability and result delivery follow Telegram ingress;
• todo visibility follows the todo plugin;
• todo dispatch requires execution plus todo;
• Typefully visibility follows the typefully plugin;
• Typefully dispatch requires execution, typefully, and Typefully external;
• voice/photo require ingress plus their individual switch.

Gates run before proposal approval, before lease, and after lease/before
dispatch. Disabling a capability preserves durable state, never interrupts an
already dispatched effect, and never makes an unknown effect retryable.
Typefully preview with external execution off permits revise/cancel/discard but
has no active approval or queue path.

Required rework 1 — truthful heartbeat monitoring

Keep the existing three fixed-key readiness heartbeat rows and the existing
three metric/alarm names. Treat the metric values as scheduled pulse-presence
signals; readiness continues to calculate stored heartbeat age separately.

Use this minimal design:

1. Add one rate(2 minutes) scheduled event to
   ConversationalExecutionWorkerFunction, enabled only by
   ConversationalExecutionIsEnabled, with a distinct fixed action such as
   conversational-execution-health-pulse.
2. The worker handler recognizes that action before stream/recovery work. It
   writes only the fixed execution_worker heartbeat and emits exactly
   ExecutionWorkerHeartbeatAgeSeconds = 0; it does not query/lease/recover an
   attempt, call a provider, retrieve a provider secret, or inspect queue
   traffic.
3. The existing recovery schedule remains rate(2 minutes) and emits exactly
   RecoveryHeartbeatAgeSeconds = 0 only after its scheduled invocation reaches
   the successful pulse point. Idle recovery is healthy.
4. The existing result-dispatcher schedule remains rate(1 minute) and emits
   exactly ResultDispatcherHeartbeatAgeSeconds = 0 after a successful
   bounded run. An empty outbox is healthy.
5. Only these three heartbeat alarms change to
   TreatMissingData: breaching. Retain Maximum, threshold > 300, 5-minute
   periods, two evaluation periods, and two datapoints to alarm. Successful
   fixed-zero pulses remain non-breaching; two missing periods can page a
   stopped schedule/function.
6. All other 14 alarms retain TreatMissingData: notBreaching. Disabled
   execution/ingress conditions still remove their heartbeat alarm and disable
   the related schedules, so an intentionally disabled component cannot page.

Do not add metric math, a monitoring Lambda, a fourth heartbeat alarm, another
metric namespace, or queue-traffic heuristics.

The new SAM schedule event will produce an additional EventBridge rule and
Lambda permission in the transformed template. Freeze their exact logical IDs,
condition/state, target, input, and function ownership. Gate A's accepted
deploy role already covers stack-prefixed EventBridge rule lifecycle/tagging
and worker Lambda Add/RemovePermission; do not add those actions to #131's
observability policy. #131 must add structural coverage proving that reuse.

Required rework 2 — bounded ingress-off maintenance

After webhook authentication but before JSON parsing:

• measure the raw update in UTF-8 bytes using the same 256 KiB maximum as
  the enabled adapter;
• reject an oversized update with 413;
• attempt no Telegram reply, conversation/model/media/plugin/domain/provider
  work, and do not parse the oversized body.

Every ingress-off maintenance or private-chat-redirect sendMessage attempt
must:

• use one fixed deadline no greater than 5 seconds through an AbortSignal;
• make at most one attempt and never retry an unknown send outcome;
• swallow timeout/network failure into the normal safe Telegram webhook
  acknowledgement so Telegram is not invited to repeat a possibly delivered
  reply;
• log only an allowlisted fixed event/error code, never token, URL, chat/update
  content, body, header, or stack trace.

Add deterministic tests with a non-returning fetch that observes abort within
the configured deadline and an oversized update that receives 413 with zero
reply/downstream calls. Retain positive proofs for authentication, private
maintenance, directed-group redirect, and at most one fixed reply.

Permanent Telegram and safety behavior

The legacy mutating Telegram fallback remains removed/unreachable. In all
states:

• /todo and /social are static guidance and never write directly;
• /podcast states that podcast creation is outside this MVP;
• /start, /help, /status, session controls, private identity, directed
  group redirect, unsupported-document failure, and reviewed media behavior
  remain as accepted;
• ingress off authenticates and performs only the bounded maintenance behavior
  above;
• rollback never restores a legacy handler or polling.

Readiness remains admin-only, exact-read/query-bounded, scan-free,
mutation-free, provider/secret-free, and redacted. Telemetry remains
fixed-cardinality and recursively safe.

Frozen application graph after rework

Refreeze from fresh source/build/transforms. The intended observability family
remains:

• one ConversationalRolloutAlarmTopic, encrypted with alias/aws/sns, ARN
  output only, no stack subscription/topic policy/contact;
• exactly three new 30-day retained custom log groups wired with
  LoggingConfig;
• exactly 17 named alarms and one topic action;
• the three heartbeat alarms use TreatMissingData: breaching; the other 14
  use notBreaching;
• no dashboard, composite, anomaly, canary, metric/filter/subscription, or
  additional observability resource.

The intended schedule family is:

• existing execution recovery schedule;
• new execution worker health-pulse schedule;
• existing result dispatcher schedule;
• their exact SAM-generated EventBridge rules and Lambda permissions.

The replacement #128 source and transforms, not the old #131 fixture, are
authoritative.

Acceptance criteria

• One immutable snapshot exposes exactly six controls, rejects invalid
  combinations, and keeps all defaults off.
• Plugin/external gates and durable todo/Typefully behavior retain the
  accepted #126/#127 boundaries.
• Legacy direct task/social/podcast/fallback mutation is unreachable and
  slash compatibility remains static.
• Ingress-off authenticates, enforces a 256 KiB pre-parse byte limit, and
  performs only one deadline-bounded maintenance/redirect attempt.
• Maintenance timeout/abort returns safely without retry, downstream work,
  or sensitive logging.
• Worker health has an execution-conditioned two-minute no-op pulse
  independent of stream/queue traffic.
• Recovery and dispatcher use their existing schedules as idle-safe
  successful pulse sources.
• Successful pulses emit fixed zero; the three heartbeat alarms alone use
  missing-as-breaching; disabled components do not alarm.
• Deterministic tests prove healthy/idle, one-period gap, stopped/two-period
  missing, disabled, handler-error, and no-work states for all three
  components.
• Readiness stays fixed-key/bounded/redacted and reports disabled
  components as not_applicable.
• Telemetry remains fixed-cardinality and drops unsafe values without
  breaking business behavior.
• Fresh transforms contain one topic, three retained groups, 17 corrected
  alarms, the new worker pulse rule/permission, existing schedules, and no
  prohibited family.
• Corrected #131 exactly matches the replacement graph and provides only
  proven tag-on-create plus existing lifecycle permission.
• The Playwright global setup passes these six explicit canonical dark values
  to its spawned test server, independent of the parent environment:
  CONVERSATIONAL_TELEGRAM_INGRESS_ENABLED=false,
  CONVERSATIONAL_EXECUTION_ENABLED=false,
  CONVERSATIONAL_ENABLED_PLUGINS=none,
  CONVERSATIONAL_TYPEFULLY_EXTERNAL_EXECUTION_ENABLED=false,
  CONVERSATIONAL_TELEGRAM_VOICE_ENABLED=false, and
  CONVERSATIONAL_TELEGRAM_PHOTO_ENABLED=false.
• Playwright readiness probes GET /api/health and resolves only on HTTP
  200; connection errors, timeouts, and every non-200 response remain
  unready until the bounded startup timeout fails clearly.
• An automated regression unsets all six controls in the parent process,
  starts the E2E server through the real bootstrap, and proves the child is
  healthy with the six dark defaults rather than inherited ambient values.
• CI=true npm --prefix backend run test:e2e completes the full suite with
  exit code 0; retries must not mask a server-startup/configuration failure.
• Production fail-closed validation is unchanged: missing, malformed, or
  contradictory rollout controls still fail outside the test-only E2E
  bootstrap, with no fallback defaults added to deployed handler/runtime
  parsing.
• Gate A and corrected Gate B are effectively applied before #128
  commit/merge/push.
• Normal CI deploys the exact all-off controls through OIDC, reaches terminal
  stack success, realizes the accepted conditional dark graph, completes
  runtime seed, and passes deployed smoke without a manual app/Lambda deploy.
• #166 A/B/C, #182's final no-replacement Cards preflight, D, and cleanup
  complete; canonical writers are reopened, temporary phase machinery is
  deleted, and a following ordinary all-off deployment succeeds before any
  live #128 canary.
• [HUMAN] Private preflight proves exactly one DataOps webhook owner, no
  poller/second registrar, expected secret-token contract, bounded pending/
  error state, and a confirmed out-of-band alarm-topic subscription.
• [HUMAN] Dark verification proves authenticated maintenance, readiness,
  unrelated portal/API health, disabled schedules/zero active alarms, and
  zero assistant/provider/domain mutation or secret retrieval.
• [HUMAN] Text + todo stage passes propose/revise/approve, duplicate-approval
  convergence, exactly one actor-owned Task, result delivery, static /todo
  nonmutation, and the accepted #126 privacy/proof boundaries.
• [HUMAN] Voice and photo stages pass separately, including use/correct/
  discard, media-off zero-download/provider-call behavior, safe cleanup
  outcomes, and the accepted alarm evidence without publishing paths/media.
• [HUMAN] Typefully preview with external execution off permits propose/
  revise/cancel but exposes no approve/queue path; no durable canary is left
  able to dispatch later.
• [HUMAN] Typefully effect creates exactly one accepted typed-public-source
  saved draft under the private account mapping, proves unscheduled/
  unpublished/unshared result semantics and conservative ambiguous-outcome
  reconciliation, and never replays an unknown effect.
• [HUMAN] Compatibility, queue/heartbeat/readiness, result-delivery, and all
  accepted alarm canaries pass with bounded sanitized observation; static
  /social and /podcast never mutate.
• [HUMAN] Reverse-order switch rollback completes only through normal OIDC,
  preserves durable proposal/attempt/outbox state, revives no expired
  presentation, restores no legacy mutation/poller, and performs no purge or
  manual Lambda/IAM edit.
• Sanitized outcomes are cross-posted to #121 and #124-#127; close each only
  when its own live criteria pass, close #128 after observation/rollback, and
  close #122 last.

Required automated verification

Run and record exact exit codes/counts:

npm --prefix backend test
npm --prefix backend run typecheck
npm --prefix backend run build
npm --prefix backend run test:telegram-transaction
npm --prefix backend run test:execution-transaction
npm --prefix backend run test:todo-transaction
npm --prefix backend run test:typefully-transaction
uv run --with pytest python -m pytest -q tests/infra
make sam-validate
AWS_CONFIG_FILE=.tmp/aws-empty/config \
AWS_SHARED_CREDENTIALS_FILE=.tmp/aws-empty/credentials \
AWS_EC2_METADATA_DISABLED=true \
AWS_DEFAULT_REGION=eu-west-1 \
sam validate --lint --template-file infra/template.full.yaml
make sam-build
git diff --check

Also require:

• focused handler tests for the worker pulse, recovery pulse, dispatcher pulse,
  256 KiB boundary, oversized rejection, abort deadline, zero retries, and zero
  downstream work;
• parsed alarm tests proving exactly 3 breaching heartbeat alarms and 14
  notBreaching alarms;
• fresh isolated dark/execution/ingress/voice/photo/Typefully-external
  transforms and exact schedule/rule/permission inventory;
• repeated deterministic graph generation and public-safe sentinel scans;
• Assistant and Architect re-review, then independent Tester and PM acceptance.

The repair requires the focused unset-parent bootstrap/readiness regression and
CI=true npm --prefix backend run test:e2e. Screenshots remain unnecessary
because no frontend surface changes. No real Telegram/provider/AWS action is an
agent test.

Remaining rollout sequence

Completed historical source/infra stages are not repeated: #121/#124-#128 source is integrated, #129/#130/#131 permissions are effective/closed, and the all-off deployment is proven.

1. Complete #166 A/B/C, the #182 final read-only Cards preflight, Phase D, and cleanup. Restore ordinary push-triggered OIDC deployment and record one successful all-off run with canonical writers open.
2. [HUMAN] Perform the private webhook/poller singleton and confirmed alarm-subscription preflight. Do not publish the bot token, secret URL, chat identity, pending payload, subscription endpoint, or provider configuration.
3. Reconfirm the Dark state through normal OIDC: ingress off, execution off, plugins none, Typefully external off, voice off, photo off. Verify authenticated maintenance and zero downstream effects.
4. Apply one normal OIDC update for Text + todo: ingress on, execution on, plugins todo, Typefully external off, voice off, photo off. Complete the bounded text/todo journey.
5. Apply Voice canary: same as Text + todo with voice on and photo off. After PASS, apply Photo canary with both voice and photo on.
6. Apply Typefully preview: ingress/execution on, plugins todo,typefully, external off, media only as separately accepted. Propose/revise/cancel; do not approve or leave queued work.
7. Apply Typefully effect: same state with external on only after #127's private account/source/reconciliation gates pass. Create exactly one saved draft and observe bounded result delivery.
8. Verify static compatibility commands, readiness, queues, heartbeats, and accepted alarm paths. Observe long enough for the fixed schedule/alarm semantics; record only sanitized counts/states.
9. Reverse switches through normal OIDC in this order: Typefully external, Typefully allowlist, photo, voice, todo allowlist, execution, ingress. Prove durable state remains safe and no legacy handler/poller returns.
10. Cross-post sanitized evidence to #121 and #124-#127, close the child issues whose own criteria pass, close #128 only after rollback/observation PASS, and close #122 last.

Every live stage is HUMAN-authorized and On-Call monitored. A failed stage stops progression and uses the reviewed switch rollback; it does not authorize a manual deploy, queue purge, retry of unknown effects, provider replay, or destructive cleanup.

Out of scope

Groups/shared context beyond redirect, web conversation/approval, memory,
scheduling/publishing, podcast/SOP mutation, new plugins, generic uploads,
second proposal/execution paths, extra observability families, provider
fallback, secret UI/plaintext credentials, manual app deployment/Lambda patch,
polling, queue purge, destructive state changes, and edits to source repos from
the #128 worktree.

Start with _docs/CONVERSATIONAL_AGENT_PLUGIN_ARCHITECTURE.md and the current #166/#182 sequencing state; the issue provides no specific source or test paths. Review the fresh source, build transforms, and normal all-off deployment before attempting staged HUMAN verification. Done means accepted live sandbox outcomes, cleanup, a successful ordinary all-off deployment, and completion of the existing HUMAN matrix with rollback evidence.