picatz / picatz/flowstate

Testing charter: write down what the 390 test files already believe, and decide the five things they don't

Open
#405 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design enhancement kind/umbrella testing
Dominant language
Go
Stars
9
Forks
0
Avg merge
3h 3m
Merged PRs (30d)
509

Description

Testing audit, design record. The audit's headline is that this repo's testing culture is strong and implicit: 2,323 tests whose conventions are consistent enough to be rules, enforced only by imitation. This issue is the charter pass — write down what's already true so it stays true, and settle the handful of genuinely open policy questions. Companion to the concrete gap issues: #400 (shared dev server), #401 (metrics), #402 (goldens/TUI), #403 (fuzzing/benchmarks), #404 (CLI harness).

Codify what is already the convention (observed, consistent, worth a page in CLAUDE.md or docs/TESTING.md)

  • External test packages by default; internal tests carry the _internal_test.go suffix so reaching into the package is a visible, named exception (~20 files do this deliberately).
  • Slice-of-struct tables with a name field, one t.Run per case (347 table loops, zero map tables — map iteration order has no business in test identity).
  • t.Parallel() is the default (1,312 uses), and non-parallel tests document their reason the way telemetry_test.go:39-43 and help_test.go:32 do (process globals, flag globals).
  • Test the protocol surface, not the internals: MCP through a real client session over in-memory transports, LSP through real jsonrpc2 over net.Pipe, RPC through httptest with real interceptors. This is the repo's strongest habit and deserves to be stated so new subsystems inherit it on purpose.
  • The security shapes: negative-direction isolation tests, containment shapes (%v/%+v/%#v/%s × value/struct/slice), bounds asserted as reached and not just unexceeded, the traversal-not-the-step rule. Already in CLAUDE.md; the charter's job is linking them to the named exemplars (TestListPagingReachesEveryRun, the six containment suites) so they're copied from live code.
  • The reachable-package pattern (plugins/*/reachable/) as the standard proof that a capability is reachable from a Flowfile — CLAUDE.md's reachability rule, in test form.

Decide the open questions (each small, each currently answered differently per file)

  1. Assertion policy: testify for scalars and errors, cmp.Diff + protocmp.Transform() for structs and protos. Today: 6,614 require vs 48 cmp.Diff. require.Equal on a nested proto prints two opaque blobs; cmp.Diff prints the three fields that differ. The rule is one sentence and the migration is incremental — apply it to new tests and to any assertion whose failure output someone curses at.
  2. The require.Eventually Goexit trap (documented at server/main_test.go:254, enforced nowhere): adopt one shared eventually-helper whose condition returns error, and consider a vet rule. 6,614 chances to re-learn this by hanging CI.
  3. The -short tier is 15 files deep. make test-fast still runs nearly everything, so the inner feedback loop is paying for tests it doesn't need. Policy: anything booting a dev server, building a binary, or sleeping real time consults testing.Short(); the fast tier's budget is a number (say, under 60s) the Makefile comment states so drift is visible.
  4. Coverage measurement is absent — no -coverprofile anywhere, so "which branches does nothing reach" is unanswerable today, which sits oddly beside a CLAUDE.md that says a bound nothing reaches is a bound nothing tests. Proposal deliberately modest: measure and publish (a CI artifact + a line in the job summary), no gate — gates on a ratio invite ratio-shaped tests; visibility invites the right ones.
  5. cmd/flow/internal/fragments is the one untested first-party package — smallest item, listed so the charter can say "zero untested packages" and mean it.

Explicitly not adopted (recording the negative decisions too)

  • temporaltest / in-process Temporal for tests: stays out for now — testsuite.StartDevServer + the #400 shared-server pattern covers it without taking the go.temporal.io/server dependency (#377 records that trade-off; if #377 ever embeds, tests revisit).
  • A third assertion library or BDD framework: the two-tool policy above is the whole answer.
  • Retry/flake-quarantine helpers: the repo has no flaky-test infrastructure and that is a feature — #400 and the ordering schedule attack flake at the source; a retry helper would hide the class of bug -cpu=1 exists to catch.

One meta-rule ties the charter to everything this session filed: when a bug is fixed, the test that lands with it is the kind that would have caught it — a golden for a rendering defect (#402), a transcript for a command-sequence defect (#404), a fuzz invariant for a rewriter defect (#403), a negative-direction case for an isolation defect. The audit shows the repo already believes this; the charter makes it the stated cost of closing a bug.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading CLAUDE.md and the testing references named in the issue, including server/main_test.go, Makefile, and the reachable-package examples under plugins/*/reachable/. Review the companion issues before deciding the five policies. Done means the existing conventions and explicit negative decisions are recorded in CLAUDE.md or docs/TESTING.md, with the chosen policy for each open question and the untested package addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
documentation, testing
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.