picatz / picatz/flowstate

sdk: no author-facing test kit — testing a task means launching the binary through a host or hand-building `map[string]*Value`, and neither checks the outputs, the claims, or the classification the host will read

Open
#1,497 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

engine enhancement testing
Dominant language
Go
Stars
9
Forks
0
Avg merge
3h 3m
Merged PRs (30d)
509

Description

Problem

The SDK's public surface is Main/Run, the options, DecodeInputs/EncodeOutputs/Literal, the egress and caller helpers, and the error constructors (pkg/flowstate/v1/plugin/sdk/*.go). There is no test package. An author who wants to test a task has two routes, and the first-party plugins use both:

  • Call an internal function directly with hand-built values (plugins/git/commit_push_test.go calls doCommitPush, bypassing the input gate on purpose, as its own comment says). This tests the author's Go and nothing about the contract: the descriptor bridge is never crossed, so #1456 (a repeated Commit output EncodeOutputs refuses) shipped green.
  • Build the binary and launch it through a real host (plugins/*/reachable/, 2,057 lines over internal/pluginreachtest, per #1476). This proves reachability and asserts ValidateSource only; it is in-tree tooling an external author cannot import.

Fixtures are the same story. Egress in a test means setting FLOWSTATE_EGRESS_POLICY_B64 by hand (sdk/egress_test.go); the caller identity and mode are installed by an unexported contextWithCaller; a resolved secret arrives as a plain value with no way to assert the input was declared to accept one. So the things the host enforces — outputs against output_message (#1476), the manifest's name lists against the descriptor (#1477), how a returned error classifies and whether it is retried (#184) — are exactly the things an author's test cannot see.

Desired outcome

A pkg/flowstate/v1/plugin/sdk/sdktest package, importable by any plugin module, that runs a task in-process the way the host would read it:

  • sdktest.Call(t, task, in proto.Message, opts...) (out proto.Message-typed, sdktest.Result) encodes in through the same value bridge the host uses, runs the manifest and descriptor checks sdk.Run runs, invokes Fn, decodes the outputs against the declared output descriptor and fails on any undeclared or mis-kinded field (the author-side half of #1476), and returns the classification as the host computes it: kind, retryable, retry-after, unknown outcome, code.
  • Fixtures as options: WithCaller(identity, namespace, mode), WithEgressPolicy(document) and WithDeploymentDefaultEgress(), WithSecret(input, value) (refusing an input not declared in SecretInputs, as the host would), WithScope(...), WithProgress(func(phase)).
  • sdktest.Conform(t, plugin): the manifest/descriptor coherence checks (#1477), the error round-trip (#184 gap 1: constructor → wire → host TaskError → text, message exactly once), the relaunch-stability check (a manifest that varies per call), and a stdout-hygiene check, all in-process with no binary and no socket.

The in-tree reachable/ harness stays the launch-based proof (#1333 owns extracting it); #713 owns the black-box binary conformance. This is the layer beneath both, and the one an outside author gets from go get alone.

Acceptance criteria

  • plugins/slack and one other first-party plugin move their task tests onto sdktest.Call with no loss of assertions, and #1456's shape is caught by the kit before the host is involved.
  • sdktest.Conform fails on a plugin whose SecretInputs names a field its descriptor does not have, and on an Fn that returns an output field its descriptor does not declare.
  • The chapter-two plugin in PLUGINS.md gains a test file using the kit, and the page shows it.
  • The kit has no dependency on internal/ packages, so it is usable from a module outside this repository (a CI leg proves it, the same one #1496 asks for).

Adjacent: #1333, #713, #1476, #1477, #184, #1341 (concurrency contract the kit can exercise by calling Fn from two goroutines).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/flowstate/v1/plugin/sdk/*.go, especially sdk.Run and the existing egress tests, then compare the reachable/ harness and internal/pluginreachtest behavior. Trace the value bridge, manifest and descriptor checks, and error classification before planning sdktest.Call and sdktest.Conform. Done means the stated acceptance criteria pass for plugins/slack, another plugin, the chapter-two example, and an external module without internal/ dependencies.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
developer-experience, testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.