picatz / picatz/flowstate

plugins: a `grpc` plugin — one unary call against reflection or a pinned descriptor set, the `buf curl` shape the ecosystem has no answer for

Open
#2,035 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
9
Forks
0
Avg merge
3h 3m
Merged PRs (30d)
509

Description

Deferred from #2021, which landed six plugins (oci, scim, ssh, docker, jose, oidc) and deliberately left this one out.

Problem

A workflow that has to call a gRPC service today has no way to do it. http speaks HTTP/1.1 and JSON; gRPC is HTTP/2 with length-prefixed protobuf framing, and the message a service takes is a schema the workflow does not hold. Every deployment that runs Connect or gRPC services internally — which is the shape this repository itself is built in — reaches them from a Flowfile only by putting a shell script in front of them.

buf curl is the reference for what this should feel like: name a service and method, supply the request as JSON, get the response as JSON, with the schema resolved either from the server's reflection service or from a descriptor set the operator pinned.

Shape

One task, grpc.call, unary only:

  • Schema from one of two places. Server reflection (grpc.reflection.v1.ServerReflection), or a FileDescriptorSet an operator configured — the pinned form being the one a deployment uses when it does not want the server deciding what its own messages mean. A workflow supplies neither.
  • Unary only. Streaming has no bounded result to put in durable history, and a step that returns "the first N messages of a stream" is a reporting limit standing in for a work limit (invariant 5).
  • Bounded in every direction: request size, response size derived from flowstatev1.MaxTaskOutputBytes the way #2021's docker and oci ceilings now are, descriptor-set size, reflection-response size, call deadline.
  • The deployment's egress policy governs the dial, per plugins/ssh and plugins/docker's remote daemon in #2021 — which may mean a grpc scheme beside ssh and postgres in netpolicy, or reusing https since the transport is HTTP/2 over TLS. Worth deciding rather than assuming.
  • Credentials through the secret boundary, never a header a Flowfile composes.

Open questions

  • Reflection is a capability: a server that offers it will describe every service it has. Is naming the method enough of a bound, or does the operator grant a service allowlist the way ssh grants commands?
  • Connect, gRPC and gRPC-Web are three wire formats over the same schema. One task with a protocol input, or the Connect client's own negotiation?
  • Does the descriptor set belong in the operator's environment (--plugin-env grpc=..., the surface #2021 added) or in a grant file like docker's and ssh's?

Acceptance

  • grpc.call with reflection and with a pinned descriptor set, both covered by a reachable/ test that launches the real binary against a real in-process gRPC server rather than a stub at the client boundary.
  • A worked example under examples/plugins/grpc/ that the corpus validates, listed in examples/README.md.
  • Every bound stated in the plugin's doc.go and asserted in a test, including the negative direction.
  • Egress governed on the actual dial path, with a test proving a denied destination is never reached.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing plugins/ssh and plugins/docker behavior, the netpolicy surface, and the changes from #2021. Resolve the open questions around protocol, reflection bounds, credentials, and descriptor-set configuration before implementation. Done means reachable/ tests cover reflection and pinned descriptors against a real gRPC server, examples/plugins/grpc/ is validated and listed, doc.go states every bound, and denied egress is tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
backend-api-design, documentation, networking, security, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.