Testing charter: write down what the 390 test files already believe, and decide the five things they don't
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 9
- Forks
- 0
- Avg merge
- 3h 3m
- Merged PRs (30d)
- 509
Description
Testing audit, design record. The audit's headline is that this repo's testing culture is strong and implicit: 2,323 tests whose conventions are consistent enough to be rules, enforced only by imitation. This issue is the charter pass — write down what's already true so it stays true, and settle the handful of genuinely open policy questions. Companion to the concrete gap issues: #400 (shared dev server), #401 (metrics), #402 (goldens/TUI), #403 (fuzzing/benchmarks), #404 (CLI harness).
Codify what is already the convention (observed, consistent, worth a page in CLAUDE.md or docs/TESTING.md)
- External test packages by default; internal tests carry the
_internal_test.gosuffix so reaching into the package is a visible, named exception (~20 files do this deliberately). - Slice-of-struct tables with a
namefield, onet.Runper case (347 table loops, zero map tables — map iteration order has no business in test identity). t.Parallel()is the default (1,312 uses), and non-parallel tests document their reason the waytelemetry_test.go:39-43andhelp_test.go:32do (process globals, flag globals).- Test the protocol surface, not the internals: MCP through a real client session over in-memory transports, LSP through real jsonrpc2 over
net.Pipe, RPC throughhttptestwith real interceptors. This is the repo's strongest habit and deserves to be stated so new subsystems inherit it on purpose. - The security shapes: negative-direction isolation tests, containment shapes (
%v/%+v/%#v/%s× value/struct/slice), bounds asserted as reached and not just unexceeded, the traversal-not-the-step rule. Already in CLAUDE.md; the charter's job is linking them to the named exemplars (TestListPagingReachesEveryRun, the six containment suites) so they're copied from live code. - The reachable-package pattern (
plugins/*/reachable/) as the standard proof that a capability is reachable from a Flowfile — CLAUDE.md's reachability rule, in test form.
Decide the open questions (each small, each currently answered differently per file)
- Assertion policy: testify for scalars and errors,
cmp.Diff+protocmp.Transform()for structs and protos. Today: 6,614requirevs 48cmp.Diff.require.Equalon a nested proto prints two opaque blobs;cmp.Diffprints the three fields that differ. The rule is one sentence and the migration is incremental — apply it to new tests and to any assertion whose failure output someone curses at. - The
require.EventuallyGoexit trap (documented atserver/main_test.go:254, enforced nowhere): adopt one shared eventually-helper whose condition returnserror, and consider a vet rule. 6,614 chances to re-learn this by hanging CI. - The
-shorttier is 15 files deep.make test-faststill runs nearly everything, so the inner feedback loop is paying for tests it doesn't need. Policy: anything booting a dev server, building a binary, or sleeping real time consultstesting.Short(); the fast tier's budget is a number (say, under 60s) the Makefile comment states so drift is visible. - Coverage measurement is absent — no
-coverprofileanywhere, so "which branches does nothing reach" is unanswerable today, which sits oddly beside a CLAUDE.md that says a bound nothing reaches is a bound nothing tests. Proposal deliberately modest: measure and publish (a CI artifact + a line in the job summary), no gate — gates on a ratio invite ratio-shaped tests; visibility invites the right ones. cmd/flow/internal/fragmentsis the one untested first-party package — smallest item, listed so the charter can say "zero untested packages" and mean it.
Explicitly not adopted (recording the negative decisions too)
- temporaltest / in-process Temporal for tests: stays out for now —
testsuite.StartDevServer+ the #400 shared-server pattern covers it without taking thego.temporal.io/serverdependency (#377 records that trade-off; if #377 ever embeds, tests revisit). - A third assertion library or BDD framework: the two-tool policy above is the whole answer.
- Retry/flake-quarantine helpers: the repo has no flaky-test infrastructure and that is a feature — #400 and the ordering schedule attack flake at the source; a retry helper would hide the class of bug
-cpu=1exists to catch.
One meta-rule ties the charter to everything this session filed: when a bug is fixed, the test that lands with it is the kind that would have caught it — a golden for a rendering defect (#402), a transcript for a command-sequence defect (#404), a fuzz invariant for a rewriter defect (#403), a negative-direction case for an isolation defect. The audit shows the repo already believes this; the charter makes it the stated cost of closing a bug.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading CLAUDE.md and the testing references named in the issue, including server/main_test.go, Makefile, and the reachable-package examples under plugins/*/reachable/. Review the companion issues before deciding the five policies. Done means the existing conventions and explicit negative decisions are recorded in CLAUDE.md or docs/TESTING.md, with the chosen policy for each open question and the untested package addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- documentation, testing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100