picatz / picatz/flowstate

testing: no tier kills a real worker mid-activity, restarts the dev server under a waiting run, or kills a plugin mid-execute; the durability claim is proved by replay and simulation, never by a process dying

Open
#1,745 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

engine enhancement testing
Dominant language
Go
Stars
9
Forks
0
Avg merge
3h 3m
Merged PRs (30d)
509

Description

Observed behavior

At eb8172f, durability is proved three ways: the conformance corpus runs the same cases on both drivers, the replay corpus checks the interpreter against recorded histories, and pkg/flowstate/v1/dst explores schedules of the local driver under seeds. #477 designs fault injection and crash-resume equivalence as slices of the simulation tier, and it says plainly that the Temporal side is "the replay tier": the replayer checks code against history and "does not and cannot ... drive our fault space".

Nothing in the tree kills a process. grep -rn 'SIGKILL\|Process.Kill' --include='*_test.go' pkg cmd finds the dev-server supervisor's own teardown and the plugin host's tests of a plugin that exits, not a test that kills a worker while an activity is in flight and asserts the run finishes on a second worker. deep.yml has no such job. The README's promise, "any workload that must finish correctly despite crashes, network failures, and long waits", is therefore held by argument (Temporal does this) plus the simulation of our own engine, and never by the event the sentence names, on the real substrate, with our real worker, plugins, secrets cache, egress policy, and Continue-As-New in the path.

The gaps this class of test finds are the ones neither tier can: a worker that holds a plugin child process which does not die with it (#1679), a secrets cache re-resolving on the new worker, an http activity retried after the first worker died between sending and recording, an undo registered on the dead worker, an audit record for a decision that never reached its sink because the process died first (--audit-required).

Desired outcome

  • internal/chaos (test-only): drives flow server dev, two flow worker processes with a plugin, and a run corpus of long activities, wait_for_signal, for_each with concurrency, undo:, and a loop that Continues-As-New. Faults, each named and seeded: SIGKILL a worker during an activity, SIGKILL the plugin during Execute, SIGSTOP a worker past the heartbeat timeout then SIGCONT, restart the dev server (history persists in its SQLite) under a waiting run, and drop the egress route for one attempt. After each fault the assertion is the conformance one: outputs, undo order, and timeline are identical to the fault-free run, and the audit trail has one record per decision.
  • Runs in deep.yml weekly, bounded like the other deep jobs, filing an issue on a divergence with the seed and the fault named. The kill-by-PID rule applies: the harness kills the PIDs it started, never by pattern.
  • flow debug replay accepts the history the chaos run produced, so a divergence is debuggable offline.

Acceptance criteria

  • The job passes on main with every fault above enabled.
  • Introducing a deliberate defect (a sync.Once guarding undo registration per process) makes the worker-kill case fail with the step named.
  • docs/ARCHITECTURE.md's durability section links the job as the evidence for its claim.

Constraints and dependencies

  • Adjacent: #477 (simulation tier; this is the third leg it describes as out of its scope), #1679 (plugin process limits), #1736 (upgrade rehearsal, which shares the harness), #1738 (attaching to a running dev server), internal/temporaltest.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with #477, internal/temporaltest, and the proposed internal/chaos harness, then inspect deep.yml and the flow server, worker, plugin, and replay entry points. Done means every seeded fault preserves outputs, undo order, timeline, and audit records, runs weekly in deep.yml, supports offline flow debug replay, and is documented in docs/ARCHITECTURE.md.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
ci-cd, devops, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
34/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.