proto: evaluate protobuf-go's opaque API — the 5,334 getter reads already port for free, the cost is 1,554 composite literals, and `API_HYBRID` makes the step reversible

Observed behavior

At 29ed8f5, every generated message in this repository uses protobuf-go's open API: exported struct fields, pointers for presence, and composite literals as the construction idiom. buf.gen.yaml passes only paths=source_relative to buf.build/protocolbuffers/go:v1.36.11, so api_level is the default (API_OPEN). The opaque API has been available since protobuf-go v1.36.0 and this module is on v1.36.12, so the choice is currently being made by omission rather than on the merits.

This is not a defect report. It is a request to decide deliberately, because the migration only gets more expensive as the schema and its call sites grow, and because the numbers below are much better than the usual "rewrite everything" intuition suggests.

Evidence

Measured at 29ed8f5, over the 658 hand-written non-test, non-generated .go files (excluding *.pb.go and *.connect.go):

So roughly three quarters of the contact surface — every read — needs no edit at all. The work is concentrated in construction, which is exactly what google.golang.org/protobuf/cmd/open2opaque rewrites mechanically.

Desired outcome

A recorded decision, backed by a measurement on this repository rather than on the upstream blog's benchmarks, on whether to adopt the opaque API and in what order.

The properties worth measuring here, rather than assuming:

• Memory. Presence becomes bit fields rather than a pointer per optional scalar. A *Workflow is held per in-flight admission and bounded by MaxSpecBytes; the engine holds specs across a run. Whether that is a rounding error or a real reduction is an empirical question this repository can answer with the benchmarks it already has.
• Aliasing and copying hazards. The open API permits *a = *b on a generated struct and unkeyed composite literals; the opaque API makes both impossible to express. walkEmbeddedWorkflows mutates current.ResolvedPlugins on every workflow it visits, and specifications are built in-process on the RPC path as well as compiled from Flowfiles, so "two positions hold the same pointer" is expressible today.
• Lazy decoding. Opaque messages can decode submessages lazily. The listing and validation paths read a small part of a large specification, which is the shape that benefits.

Acceptance criteria

• A benchmark on the largest example (enterprise-fund-transfer) comparing open and opaque generation for compile, CheckSpecSize/validation, and one admission path, reported as allocations and bytes rather than as an impression.
• A decision recorded on the issue: adopt, adopt-later with a named trigger, or decline with the reason.
• If adopted: api_level=API_HYBRID lands first as a buf.gen.yaml change plus a regeneration, with no hand-written call site touched. Hybrid generates both surfaces, so it is reversible by reverting one line and regenerating, and it lets the open2opaque conversion proceed package by package rather than as one sweep — the same rule #521 records for the Go modernizers, for the same reason.
• If adopted: the per-file (pb.go).api_level option is used to move one .proto at a time, and tools/gate's generated-artifact check plus the genguard hook keep hand edits out (invariant 9).

Constraints and dependencies

• Invariant 1 (proto-first) and invariant 9 (generated artifacts are derived) both favour making this a generator setting rather than a hand-written convention; nothing about the migration is a hand edit.
• ConnectRPC, protovalidate, and the plugin protocol all consume descriptors rather than the Go struct shape, so they are unaffected by api_level. This wants confirming rather than assuming, since buf.build/go/protovalidate reads descriptors at runtime.
• The six plugin modules under plugins/ generate their own messages from their own buf.gen.yaml; they can move independently or not at all, and the SDK boundary between them and the host is protobuf bytes, not Go types.
• internal/schemaifacepilot reads protoreflect descriptors and one GoField name per selection (#1227); its Selection.GoField is the one place a Go field name is written down, and it would need the setter name instead.

Open questions

• Is the memory win material at this repository's specification sizes, or is it below the noise of everything else an admission spends?
• Does hybrid's doubled generated output (29k lines is already the largest generated surface here) cost more in review and repository size than the eventual opaque win returns?
• Should the plugin modules follow the root module, or is a deliberate split acceptable given the boundary between them is wire bytes?

Start with buf.gen.yaml, the existing benchmarks for enterprise-fund-transfer, and the generated .pb.go files; inspect tools/gate and internal/schemaifacepilot for migration constraints. Compare open and opaque generation on compile, validation, and one admission path, reporting allocations and bytes. Done means the measurements are recorded with an adopt, defer, or decline decision; adoption also documents the hybrid setting and regeneration path.