Foundations record: data lifecycle, operational reality, project trust, adoption mechanics
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 9
- Forks
- 0
- Avg merge
- 3h 3m
- Merged PRs (30d)
- 509
Description
Fifth entry in the record set (#336, #337, #341, #346), and it revises one thing in the dispatch-policy design (#326's neighborhood): decision recording, below. The other records cover what Flowstate does; this one covers what it is: a data controller, an operated system, a security-accountable project. The organizing observation: durable execution's superpower is that nothing is forgotten, and privacy law, operations, and procurement all demand the opposite in places. Things must be forgettable, restorable, exportable, and accountable. Several items are urgent precisely because they are cheap now and prohibitive later: an encryption seam before the first production history exists is a design decision; after the thousandth, it is a migration program.
Principles these workstreams add
- Forgettable by construction. Sensitive material enters history only through the encryption seam, keyed so that erasure is key deletion. References-not-material (secrets, credentials, artifacts) stays the first line; the codec is the line for what must flow through state anyway.
- Decisions are recorded evidence. A policy verdict, allow or deny, is an event carrying the rule and the evaluated fact values that produced it. A transcript that shows only refusals cannot answer "why was this permitted", which is the question auditors actually ask.
- The project meets its own bar. Every standard our docs urge on deployments (signed artifacts, provenance, disclosure policy, threat model, honest limits) is one the project itself visibly meets first.
- Scaffolds never claim fidelity. An importer produces an honest starting point with explicit TODO markers where semantics do not map. A wrong-but-running conversion is worse than a marked gap.
- Reversibility is a feature tier. Backup, restore, export, and leave-the-deployment paths are designed capabilities with tests, not emergent properties of the persistence layer.
Workstreams
A. Data lifecycle: the encryption seam, crypto-shredding, retention, residency. The substrate never encrypts payloads server-side; the seam is the client's converter chain, where a payload codec runs outside the workflow sandbox and may hold keys. Four coupled pieces. (1) The codec slot, now: wired into both drivers' client construction identically (both-drivers applies to ciphertext too), null codec by default, interface stable from the first release anyone runs in production; codec implementations as plugins, with key custody unified with flow keys and the issuer work rather than a second key story. Failure-path encoding ships enabled whenever a codec is configured, because error strings carrying context around an otherwise-encrypted history is the fail-open composition shape we already know. (2) Crypto-shredding as a product feature: key per tenant at minimum, key per declared subject where a workflow names one; flow shred destroys keys and records the destruction as an attested administrative event, the audit trail of forgetting without the forgotten content. Open questions: key hierarchy (subject keys wrapped by tenant keys), what a live run whose payloads are shredded mid-flight does (fail closed and say why), and an audit of every place payload bytes could escape the seam (search attributes, visibility, logs, spans). (3) Retention as declaration: per-workflow retention mapped onto namespace retention, artifact GC (#341 A), and policy-window horizons, one coherent story with three consumers. (4) Residency as topology, v1 as validation only: a workflow declaring residency a deployment cannot satisfy is a load error; no replication machinery.
B. The deployment gradient and reversibility. The individual-and-small-team end currently has no home between flow run local and operating a cluster, and adoption dies in exactly that gap. (1) A blessed small profile: one host, service units, single-file or single-database persistence with streaming-replication guidance, server and worker and substrate co-resident, honest limits stated the way the shared-namespace warning already models (what breaks first, when to graduate, how to migrate up). Packaging and documentation work, not a third driver; no strain on both-drivers-agree. (2) BCDR as designed capability: backup and restore runbooks with tests for the small profile, and flow export / flow import for runs and artifacts (versioned schema, breaking-checked per #346 B) so leaving a deployment is a documented path.
C. The security program as artifact set. (1) SECURITY.md and a disclosure policy now: contact, scope, safe harbor, response targets. (2) Signed, reproducible releases with provenance, verified in CI; the long game is releases built by Flowstate runs carrying run identity as build identity (#337). (3) THREAT_MODEL.md: the composed adversary analysis not yet written anywhere, including prompt injection through the transparent agent loop (#341 invariant 2's postures, per-step tool bounds, plan-batch approval, egress, composed against a stated attacker), the policy file as crown jewel and the governance recursion (policy changes shipped as Flowstate runs with attested approvals), webhook ingress, the issuer as a single point of failure, and per-tier sandbox assumptions (#341 E).
D. Decision recording and retry semantics. Revision to the dispatch-policy design. (1) Record allows, not only denials, per principle 2: every verdict persists its rule and evaluated fact values, which is what makes flow policy explain (#341 I) a transcript rather than a guess. (2) Choose retry semantics explicitly: a step denied inside a sliding window and retried after it slides will pass; re-decision per attempt and verdict pinning are both defensible, undecided is not. Proposed default: re-decide per attempt (a rate limit that never unblocks is the worse surprise), transcript marking re-decisions, per-rule opt-in to pin where the intent is "this run was refused, full stop". Lands with the dispatch-policy implementation, with a virtual-clock flow test case proving whichever is chosen.
E. Fault injection in flow test. The thing durable execution is for, exercised nowhere: crash between steps, did compensations fire, did the saga converge. Scripted faults in the existing test grammar: fail an activity's first N attempts, crash and resume after a named step, time out a specific wait, poison one item of a fan-out. All through the local driver, deterministic and sub-second under the virtual clock. The differentiated claims are the declarative scenario format, crash-and-resume without infrastructure, and making undo: paths testably mandatory (a lint flagging compensations no test ever executes is the workflow analogue of uncovered error paths).
F. Importers as scaffolds. flow import for existing CI workflow files, producing structure where mappings are honest and loud TODO blocks where semantics diverge, never a fidelity claim (principle 4). Coexistence via federation (#343's characterized path) means import is additive: trigger from CI first, convert one stage, or convert wholesale. Cheap, and likely worth more first-afternoon adoption than any single feature in the other records. A crontab importer is the trivial sibling.
G. Small standing decisions, each expensive only late. (1) Telemetry policy before telemetry: publish the posture (none, or opt-in with a published schema) while it is trivially true. (2) Model pinning as supply chain: an llm step's model identifier is a dependency; pin it like a plugin digest, record it in the frozen spec, surface provider-side deprecations as diagnostics. A provider silently swapping models under a frozen spec is a reproducibility hole specific to the agent era. (3) Accessibility from fragment one: keyboard navigation and screen-reader semantics as acceptance criteria for every shipped fragment (#350 onward), WCAG 2.2 AA as the stated bar, because retrofitting across an ecosystem of embedded fragments is the miserable version.
Sequencing
- A.1, the codec seam: the convex-cost item; the interface exists before anyone's production history does.
- C.1 and G.1: documents, days not weeks, timing-sensitive.
- D with the dispatch-policy implementation, not after it.
- B.1 with C.2 alongside as release engineering.
- E with or immediately after the first
undo:-heavy example. - A.2 through A.4, B.2, C.3, F, G.2, G.3 by demand signal, none deferred past the first enterprise conversation.
Anti-goals
No multi-region machinery in v1 (residency is declaration plus validation). No fidelity-claiming importers and no compatibility mode: the scaffold converges to idiomatic Flowstate or it is not finished. No compliance theater: the threat model documents real boundaries and real gaps in the same register as the README's honesty notes. No second key-management story: codec keys, signing keys, and issuer keys are one custody design or the design is wrong. No telemetry before the policy, no policy after the telemetry.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Treat this as a roadmap rather than a single starter task. Begin with the sequencing section and select one bounded workstream, such as SECURITY.md, THREAT_MODEL.md, or the codec seam in both drivers' client construction. Read the named entry points and existing flow test grammar first; done means meeting that workstream's stated acceptance criteria and adding the specified tests or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cli, devops, documentation, security, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 22/100