picatz / picatz/flowstate

analysis/analysistest: expectations live in the fixture — `# want code "message"` comments in a Flowfile, a `.golden` file for the file after `edits`, and one `Run(t, dir, analyzers...)` so a check's positive, negative and fix directions are proved by files under `testdata/` rather than by Go tables that restate the message

Open
#1,602 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dsl enhancement testing
Dominant language
Go
Stars
9
Forks
0
Avg merge
3h 3m
Merged PRs (30d)
509

Description

Observed behavior

The validator's tests are Go tables asserting line, column and message text (flowfile/parse_test.go, validate_test.go; DSL.md's "Executing the flattening" records that ~59 assertions moved when a key moved two columns, and that resetting them wholesale "is how a real regression gets absorbed into a migration diff"). The LSP tests assert the underlined text instead, which survives a move. Neither form lets a reader see the check and its expectation in one place, and neither exercises a suggested edit end to end: SuggestedEdit/TextChange exist on the schema and flow fix has its own round-trip test, but no harness applies a diagnostic's edits and compares the result.

analysistest.Run is the shape: the fixture carries // want "regexp" on the line the diagnostic is expected at, the harness fails on an unexpected diagnostic and on an expected one that did not fire, and RunWithSuggestedFixes applies the fixes and diffs against file.golden. Buf's bufplugin-go test kit does the same for check rules.

Desired outcome

analysis/analysistest beside the framework:

func Run(t testing.TB, testdata string, analyzers ...*analysis.Analyzer) []*Result
func RunWithSuggestedEdits(t testing.TB, testdata string, analyzers ...*analysis.Analyzer)

A fixture is a Flowfile (or a compiled spec, so source-free producers are covered) whose comments carry expectations:

steps:
  - id: fetch
    http:
      url: ${vars.base + "/v1"}   # want unresolved-reference "vars.base"

# want <code> "<regexp>" on the offending line (the position the SourceMap projects), # want alone for "some diagnostic here", and # nowant to assert silence at a line the analyzer is known to be tempted by. A diagnostic with no expectation fails the test naming the file and line; an expectation with no diagnostic fails naming the code. RunWithSuggestedEdits writes file.yaml.golden on -update and compares otherwise, through the same TextChange application the LSP's code action performs, so the CLI, the editor and the test agree on what an edit does.

Acceptance criteria

  • The three flowfile.Lint rules and the six coded validator classes each have a testdata/<name>/ directory with positive, negative and (where edits exist) golden fixtures, and their Go table tests for the same cases are deleted.
  • A fixture whose comment expects a code the analyzer does not declare fails at load, not at run, with the declared codes listed.
  • The harness runs a fixture through the CLI driver and the LSP driver and asserts identical diagnostics, which is the first test that the two hosts cannot drift.
  • docs/PLUGINS.md's analyzer chapter (with the protocol sibling) shows a plugin author running the same harness against their out-of-process analyzer.

Constraints and dependencies

  • Positions come from the SourceMap sibling (#1568); until it lands the harness uses flowfile.Positions through an internal adapter and states so.
  • # want is a comment convention read by the harness only; the compiler never sees it, so the grammar gains nothing and STYLE's fence rules are untouched.
  • Regexps in expectations are bounded and anchored the way analysistest bounds them, so a fixture cannot hang the test.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with flowfile/parse_test.go and validate_test.go, then inspect the existing SuggestedEdit/TextChange schema, the flow fix round-trip test, and the LSP code-action path. Use the SourceMap dependency (#1568) and existing analysistest shape as references. Done means the harness supports want/nowant fixtures and golden edits, migrated rule and validator tests, matching CLI/LSP diagnostics, and updated docs/PLUGINS.md.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
documentation, testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.