testing: no tier kills a real worker mid-activity, restarts the dev server under a waiting run, or kills a plugin mid-execute; the durability claim is proved by replay and simulation, never by a process dying
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 9
- Forks
- 0
- Avg merge
- 3h 3m
- Merged PRs (30d)
- 509
Description
Observed behavior
At eb8172f, durability is proved three ways: the conformance corpus runs the same cases on both drivers, the replay corpus checks the interpreter against recorded histories, and pkg/flowstate/v1/dst explores schedules of the local driver under seeds. #477 designs fault injection and crash-resume equivalence as slices of the simulation tier, and it says plainly that the Temporal side is "the replay tier": the replayer checks code against history and "does not and cannot ... drive our fault space".
Nothing in the tree kills a process. grep -rn 'SIGKILL\|Process.Kill' --include='*_test.go' pkg cmd finds the dev-server supervisor's own teardown and the plugin host's tests of a plugin that exits, not a test that kills a worker while an activity is in flight and asserts the run finishes on a second worker. deep.yml has no such job. The README's promise, "any workload that must finish correctly despite crashes, network failures, and long waits", is therefore held by argument (Temporal does this) plus the simulation of our own engine, and never by the event the sentence names, on the real substrate, with our real worker, plugins, secrets cache, egress policy, and Continue-As-New in the path.
The gaps this class of test finds are the ones neither tier can: a worker that holds a plugin child process which does not die with it (#1679), a secrets cache re-resolving on the new worker, an http activity retried after the first worker died between sending and recording, an undo registered on the dead worker, an audit record for a decision that never reached its sink because the process died first (--audit-required).
Desired outcome
internal/chaos(test-only): drivesflow server dev, twoflow workerprocesses with a plugin, and a run corpus of long activities,wait_for_signal,for_eachwith concurrency,undo:, and a loop that Continues-As-New. Faults, each named and seeded:SIGKILLa worker during an activity,SIGKILLthe plugin duringExecute,SIGSTOPa worker past the heartbeat timeout thenSIGCONT, restart the dev server (history persists in its SQLite) under a waiting run, and drop the egress route for one attempt. After each fault the assertion is the conformance one: outputs, undo order, and timeline are identical to the fault-free run, and the audit trail has one record per decision.- Runs in
deep.ymlweekly, bounded like the other deep jobs, filing an issue on a divergence with the seed and the fault named. The kill-by-PID rule applies: the harness kills the PIDs it started, never by pattern. flow debug replayaccepts the history the chaos run produced, so a divergence is debuggable offline.
Acceptance criteria
- The job passes on
mainwith every fault above enabled. - Introducing a deliberate defect (a
sync.Onceguarding undo registration per process) makes the worker-kill case fail with the step named. docs/ARCHITECTURE.md's durability section links the job as the evidence for its claim.
Constraints and dependencies
- Adjacent: #477 (simulation tier; this is the third leg it describes as out of its scope), #1679 (plugin process limits), #1736 (upgrade rehearsal, which shares the harness), #1738 (attaching to a running dev server),
internal/temporaltest.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with #477, internal/temporaltest, and the proposed internal/chaos harness, then inspect deep.yml and the flow server, worker, plugin, and replay entry points. Done means every seeded fault preserves outputs, undo order, timeline, and audit records, runs weekly in deep.yml, supports offline flow debug replay, and is documented in docs/ARCHITECTURE.md.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- ci-cd, devops, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 34/100