analysis/analysistest: expectations live in the fixture — `# want code "message"` comments in a Flowfile, a `.golden` file for the file after `edits`, and one `Run(t, dir, analyzers...)` so a check's positive, negative and fix directions are proved by files under `testdata/` rather than by Go tables that restate the message
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 9
- Forks
- 0
- Avg merge
- 3h 3m
- Merged PRs (30d)
- 509
Description
Observed behavior
The validator's tests are Go tables asserting line, column and message text (flowfile/parse_test.go, validate_test.go; DSL.md's "Executing the flattening" records that ~59 assertions moved when a key moved two columns, and that resetting them wholesale "is how a real regression gets absorbed into a migration diff"). The LSP tests assert the underlined text instead, which survives a move. Neither form lets a reader see the check and its expectation in one place, and neither exercises a suggested edit end to end: SuggestedEdit/TextChange exist on the schema and flow fix has its own round-trip test, but no harness applies a diagnostic's edits and compares the result.
analysistest.Run is the shape: the fixture carries // want "regexp" on the line the diagnostic is expected at, the harness fails on an unexpected diagnostic and on an expected one that did not fire, and RunWithSuggestedFixes applies the fixes and diffs against file.golden. Buf's bufplugin-go test kit does the same for check rules.
Desired outcome
analysis/analysistest beside the framework:
func Run(t testing.TB, testdata string, analyzers ...*analysis.Analyzer) []*Result
func RunWithSuggestedEdits(t testing.TB, testdata string, analyzers ...*analysis.Analyzer)
A fixture is a Flowfile (or a compiled spec, so source-free producers are covered) whose comments carry expectations:
steps:
- id: fetch
http:
url: ${vars.base + "/v1"} # want unresolved-reference "vars.base"
# want <code> "<regexp>" on the offending line (the position the SourceMap projects), # want alone for "some diagnostic here", and # nowant to assert silence at a line the analyzer is known to be tempted by. A diagnostic with no expectation fails the test naming the file and line; an expectation with no diagnostic fails naming the code. RunWithSuggestedEdits writes file.yaml.golden on -update and compares otherwise, through the same TextChange application the LSP's code action performs, so the CLI, the editor and the test agree on what an edit does.
Acceptance criteria
- The three
flowfile.Lintrules and the six coded validator classes each have atestdata/<name>/directory with positive, negative and (where edits exist) golden fixtures, and their Go table tests for the same cases are deleted. - A fixture whose comment expects a code the analyzer does not declare fails at load, not at run, with the declared codes listed.
- The harness runs a fixture through the CLI driver and the LSP driver and asserts identical diagnostics, which is the first test that the two hosts cannot drift.
docs/PLUGINS.md's analyzer chapter (with the protocol sibling) shows a plugin author running the same harness against their out-of-process analyzer.
Constraints and dependencies
- Positions come from the
SourceMapsibling (#1568); until it lands the harness usesflowfile.Positionsthrough an internal adapter and states so. # wantis a comment convention read by the harness only; the compiler never sees it, so the grammar gains nothing and STYLE's fence rules are untouched.- Regexps in expectations are bounded and anchored the way
analysistestbounds them, so a fixture cannot hang the test.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with flowfile/parse_test.go and validate_test.go, then inspect the existing SuggestedEdit/TextChange schema, the flow fix round-trip test, and the LSP code-action path. Use the SourceMap dependency (#1568) and existing analysistest shape as references. Done means the harness supports want/nowant fixtures and golden edits, migrated rule and validator tests, matching CLI/LSP diagnostics, and updated docs/PLUGINS.md.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- documentation, testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100