feat(policy)!: add a public policy protobuf and keep runtime policy internal

User story

As a policy author or SDK user, I want one generated policy type for files and public APIs, so Rust, Go, Python, and TypeScript agree on what users may configure while OpenShell keeps runtime-only policy state private.

Problem statement

OpenShell currently exposes openshell.sandbox.v1.SandboxPolicy through its public API. That message is also the gateway's normalized enforcement model, the stored policy payload, and the message delivered to the supervisor.

Those jobs have different authority. The runtime message contains fields such as advisor_proposed and provider_credentialed. The gateway and sandbox set those fields after accepting user input. They must not be part of an authorable SDK type.

The authored YAML has a separate Rust model. PR #3334 moves that model into openshell-policy-schema, which gives Rust one bounded parser and one authored representation. Other SDKs still cannot reuse it, and the gateway still converts between a handwritten YAML model and protobuf.

Using the current runtime message as the file schema would expose internal fields and make policy files match normalized runtime storage rather than the author-facing language. The better boundary is a public authored protobuf plus an explicit conversion to the existing internal protobuf.

Impact / why this matters

Each policy field currently needs coordinated changes across handwritten YAML types, runtime protobuf, conversions, validation, API clients, and tests. A missed conversion can silently drop security intent.

The public API also gives SDKs a type with more authority than users should have. The server can ignore or clear internal fields, but generated clients still advertise them as valid inputs.

Go, Python, and TypeScript cannot share the Rust-authored model. Any SDK that wants to read policy YAML must recreate its shape and defaults. Without shared fixtures, those implementations will drift.

Existing policy files also outlive API clients. They are checked into repositories, passed through environment configuration, and baked into images at /etc/openshell/policy.yaml. OpenShell should keep accepting those files through a compatibility path even though the new API may break.

Proposed design

Separate public intent from internal enforcement

Add a public protobuf package for authored sandbox policies, for example openshell.policy.v1. It should define only fields that a policy author may control.

Keep the existing openshell.sandbox.v1.SandboxPolicy as the internal normalized and effective representation. Internal compute-driver and supervisor protocols can continue using it. Runtime-only fields stay there.

policy YAML or generated SDK object
  -> public authored SandboxPolicy
  -> intrinsic validation
  -> checked lowering and normalization
  -> internal base SandboxPolicy
  -> provider and global composition
  -> internal effective SandboxPolicy
  -> supervisor and enforcement

The public API should use the authored package for sandbox creation, policy replacement, merge operations, revisions, and draft rules. The gateway should translate at a shared policy boundary before invoking existing operational code.

Handlers may call the translation boundary, but they should not contain field-by-field conversion code. openshell-policy should own named operations for:

• lowering an authored policy or authored rule into the internal model
• projecting an internal base policy into canonical public form
• projecting an effective internal policy into a read-only public view

The two projections are different. A base policy represents stored user intent. An effective policy may contain provider rules, advisor provenance, credential markers, and other derived state. Public responses must not present an effective projection as author input that can be submitted unchanged.

Keep current internal storage

Continue storing the normalized internal protobuf for now. This matches current behavior and avoids a database migration.

OpenShell already treats policy round trips as semantic rather than textual. YAML enters through the handwritten model, lowers into the internal protobuf, and later serializes back into canonical YAML. Comments, anchors, formatting, source ordering, shorthand provenance, and omitted defaults are not retained today.

The conversion should preserve these invariants:

project_base(lower(public_policy)) == canonicalize(public_policy)

lower(project_base(internal_base_policy)) == canonicalize(internal_base_policy)

Those invariants do not apply to arbitrary effective policies because public projection intentionally removes internal authority.

Keep the existing internal policy hash unless the API explicitly introduces a second authored-policy identity. Document policy_hash as the identity of the normalized internal policy, not a hash of the YAML source text.

Replace every public reference to internal policy messages

Changing SandboxSpec.policy is only the first step. Audit all public imports of openshell.sandbox.v1, including:

• SandboxSpec.policy
• UpdateConfigRequest.policy
• SandboxPolicyRevision.policy
• policy merge operations that use internal network rules and L7 rules
• policy-advisor draft proposed_rule
• policy-advisor draft edit requests
• current and candidate effective policies returned for review
• provider-profile endpoints and binaries that currently reuse internal policy messages

Provider profiles may reuse public authored endpoint and L7 messages when authoring authority and semantics match. The larger provider-profile YAML and resource-model refactor should remain a follow-up.

The compute-driver policy field and GetSandboxConfigResponse.policy are internal delivery paths. They should continue using the runtime message.

Use a thin YAML layer before ProtoJSON

Protobuf does not parse YAML. An SDK that promises policy-file parsing needs a YAML library even when protobuf owns the semantic schema.

The policy file path should be:

bounded YAML parser
  -> generic YAML tree
  -> YAML safety checks and shorthand normalization
  -> JSON-compatible value
  -> strict ProtoJSON decoding
  -> generated public policy message
  -> Proto validation

This layer must not introduce another handwritten policy object tree. It should walk generic YAML values, enforce syntax rules, and rewrite the few forms that standard ProtoJSON cannot express. The generated public message remains the only semantic model.

Every SDK that supports existing policy YAML will need this small layer in its own language. That is acceptable, but only with a language-neutral conformance corpus. SDKs that accept generated messages or canonical JSON only do not need a YAML dependency.

Account for YAML forms that protobuf cannot preserve directly

The current authored language is richer than standard ProtoJSON in the following places.

Authored behavior	Why protobuf or ProtoJSON is insufficient	Required treatment
Canonical policy YAML uses snake-case names such as filesystem_policy and include_workdir.	ProtoJSON emits lower-camel names by default, although parsers accept original proto field names too. Different SDK printers may choose different output options.	Define the canonical YAML spelling explicitly. Use proto field names or json_name consistently, and test output in every SDK.
version, L7Rule.allow, binary path, credential-binding provider, and middleware middleware are required.	Proto3 supplies zero values and absent messages instead of enforcing required input.	Declare requiredness with Proto validation and return field-scoped errors. Validate the supported policy version rather than accepting zero.
A query matcher may be a scalar string or an { any: [...] } object.	A protobuf field has one JSON type. A message-valued field cannot also accept a string through standard ProtoJSON.	The YAML layer expands a scalar into the public matcher message. Canonical output may choose the scalar shorthand again.
tool accepts the same scalar-or-object matcher shorthand.	A typed matcher message cannot parse the scalar form directly.	Keep tool as an authored field in the public proto. The YAML layer expands scalar values before ProtoJSON.
MCP params are recursive maps whose leaves may be scalar or any matchers.	Proto maps cannot have values that are recursively either another map or a scalar.	Define a typed recursive authored value or normalize the generic YAML tree into a canonical matcher structure before ProtoJSON. Do not use untyped values for the whole policy.
port is a scalar shorthand for a one-element ports list.	Protobuf can contain both fields, but a repeated field cannot participate directly in a oneof. It cannot enforce the authored mutual-exclusion rule by type alone.	Keep both public fields for compatibility, validate mutual exclusion, and canonicalize one port to the documented output form.
A missing filesystem_policy differs from filesystem_policy: {}. The latter explicitly sets include_workdir: false.	Proto message presence can represent this, but code that eagerly creates an empty message erases the distinction. Empty process currently canonicalizes back to omission.	Preserve message presence through the public API and lowering. Document which present-empty sections carry meaning and canonicalize semantically empty sections consistently.
Omitted mcp.versions selects the pinned default, while an explicit empty list is rejected.	Proto repeated fields do not retain presence. Omitted and empty both decode to an empty list. A YAML-only check would leave direct SDK callers unable to express the distinction.	Give versions a presence-bearing wrapper in the public proto. The YAML layer maps the existing list syntax into that wrapper. Validation rejects a present empty wrapper, and lowering materializes the default when the wrapper is absent.
Explicit null is rejected for presence-sensitive policy fields.	ProtoJSON accepts null for most fields and treats it as unset.	Reject null in the YAML layer before ProtoJSON decoding. This includes top-level optional sections, endpoint credential_binding, json_rpc, mcp, MCP versions and optional booleans, rule tool, and middleware endpoint selectors.
MCP optional booleans distinguish omitted from explicit false.	Plain proto3 booleans have no presence.	Use optional bool in the public proto. The YAML layer still rejects explicit null.
A missing network-rule or middleware name falls back to its map key.	Protobuf represents the missing string but does not know the surrounding map key should become the effective name.	Perform the fallback during lowering and return the materialized name in canonical output.
mcp.max_body_bytes and json_rpc.max_body_bytes are nested authored settings.	The internal message flattens them into json_rpc_max_body_bytes, and MCP currently wins if both stanzas are present.	Model the nested authored shape in the public proto. Reject conflicting stanzas or document the current precedence, then lower into the shared internal field.
MCP tool lowers to the runtime params["name"] matcher.	The internal message does not retain whether the author used tool or params.name. If both are supplied today, explicit params.name wins.	Keep tool first-class publicly. Define the conflict rule, lower centrally, and use the documented tool form in canonical output. Preserve current MCP method elision where it is semantic.
Nested MCP parameter paths flatten into dot-separated runtime keys.	The internal map cannot retain the original nested spelling, and some flat-key collisions cannot be expanded safely.	Validate collisions before lowering. Canonical public projection must use one documented lossless form.
Lowercase string values such as best_effort, read-only, and protocol names do not match normal generated enum JSON names.	ProtoJSON emits enum identifiers, usually uppercase prefixed names.	Either retain validated strings in the authored proto or make the YAML layer map documented strings to enums.
Authored maps currently serialize in sorted order.	Protobuf map iteration and ProtoJSON output order are unspecified.	Canonical YAML serialization must sort maps recursively. Hashing must not depend on generated map iteration order.
Canonical YAML omits most empty strings, lists, maps, false booleans, zero limits, and middleware order zero. Present filesystem and Landlock sections have their own default-emission rules.	ProtoJSON printers have their own default-emission options and cannot reproduce the current field-by-field rules by themselves.	Own canonical YAML serialization in the schema library and test each omission rule.
YAML comments, anchors, aliases, merge keys, tags, key order, quoting, and flow style have no protobuf representation.	They are document syntax, not policy data.	Bound or reject unsafe constructs. Canonical serialization does not preserve presentation. Migration must warn about this.
YAML libraries disagree on implicit scalar typing, especially between YAML 1.1 and 1.2.	A value such as on may become a boolean in one SDK and a string in another before ProtoJSON sees it.	Define one JSON-compatible scalar-resolution policy and include ambiguous values in the cross-language fixtures.
YAML permits duplicate and non-string mapping keys.	ProtoJSON objects and protobuf maps require unique string keys. Some JSON decoders silently keep the last duplicate.	Reject duplicates and non-string keys before conversion.
Authored policy objects reject unknown fields, except for the open middleware config map.	ProtoJSON libraries often expose an ignore-unknown option, and shorthand rewriting can accidentally consume a misspelled key.	Never enable unknown-field ignoring. Validate closed shorthand objects before rewriting and let descriptors validate the remaining tree.
YAML integers can exceed protobuf field ranges, and policy ports are limited to 65535 even when the proto field uses uint32.	Wire types alone do not express the policy bound.	Enforce numeric range in YAML decoding and Proto validation.
Middleware config accepts a string-keyed JSON-compatible object and permits null as user data.	google.protobuf.Struct stores numbers as doubles, does not retain YAML syntax, and needs special ProtoJSON handling in some generated libraries.	Restrict this subtree to the JSON data model, preserve null here, reject unsafe numeric values, and test precision and round trips.
The runtime policy contains advisor_proposed, provider_credentialed, and may gain more derived fields.	A direct decoder into the runtime message would make those fields appear authorable.	Exclude them from the public proto. Lowering creates only base state, and internal composition stamps derived authority later.

This is the known inventory from the current authored schema. The conformance corpus should lock each case before the handwritten model is removed.

Preserve current canonical round-trip behavior

The current API already loses the author's choice of shorthand. It stores the normalized internal protobuf, not the YAML syntax.

For MCP, tool lowers to params["name"]. When OpenShell exports YAML, the serializer recognizes that parameter and emits canonical tool syntax again. The same pattern applies to scalar glob matchers, compact single ports, nested MCP settings, and other author-facing conveniences.

The new design should preserve that behavior. It does not need to remember which equivalent spelling the author submitted. It must return a readable canonical public policy with the same meaning.

Validation ownership

Use buf.validate annotations on the public messages for portable structural and message-local constraints. Good candidates include field bounds, required presence, enum values, collection limits, formats, uniqueness, and simple field relationships.

Keep the other validation stages separate:

Stage	Responsibility
YAML front end	Resource limits, document count, duplicate keys, aliases, merge keys, tags, string map keys, explicit null, and shorthand normalization.
ProtoJSON decoder	Strict field-name and type decoding with unknown fields rejected.
Proto validation	Portable intrinsic constraints on the public message.
Lowering	Defaults, name fallback, port normalization, nested-to-flat conversion, and authority enforcement.
Internal domain validation	Policy merges, provider composition, reference checks, runtime capabilities, and effective-policy safety.

The gateway remains authoritative. SDK validation improves feedback but cannot replace validation after API ingress.

Keep complex policy semantics out of CEL. CEL is useful for small local relationships. Policy merging, provider enrichment, and checks that depend on external state belong in domain code.

SDK support

Generate the public policy package for Rust, Go, Python, and TypeScript. Each SDK should expose generated policy messages. SDKs that offer file parsing should also expose a helper such as parsePolicyYaml and canonical YAML or JSON serialization.

The YAML helpers may use different parser libraries, but they must pass the same fixtures. Compare normalized messages, error categories, field paths, and stable validation rule IDs rather than exact prose.

Official Protovalidate implementations exist for Go, Python, and JavaScript or TypeScript. Buf does not currently provide an official Rust implementation. Evaluate prost-protovalidate behind an implementation-neutral Rust validation function. If it is unsuitable, keep the annotations as the shared declaration and implement Rust validation behind the same crate-owned boundary.

Compatibility and migration

Keep accepting current policy YAML at file-loading boundaries for a documented transition period. PR #3334 can provide the bounded Rust compatibility decoder and the reference behavior for the cross-language thin YAML layers.

Provide a migration command with check mode and non-destructive output by default. It should report semantic rewrites, emit canonical public-policy YAML, parse its own output, and verify normalized equality. In-place migration must be explicit and recoverable.

The new public API may break and does not need to accept the old internal policy message. Stored policies need no format migration while the gateway continues to store the internal protobuf.

Acceptance criteria

• A versioned public protobuf package defines every author-controlled sandbox policy field and contains no runtime-derived fields.
• The existing sandbox policy protobuf remains the internal normalized and effective model used by operational policy code and supervisor protocols.
• All public API references to internal policy, endpoint, rule, and L7 messages are removed or explicitly documented as internal-only exceptions.
• Public policy requests pass through one shared, checked lowering path before internal use.
• Base-policy responses and effective-policy responses use deliberate projections with documented semantics.
• advisor_proposed, provider_credentialed, and any future derived authority cannot be supplied through the public message or policy YAML.
• The gateway continues to store normalized internal policy payloads, and existing stored policies remain readable without a data migration.
• Policy hash semantics remain stable or receive an explicit migration plan.
• openshell-policy-schema generates the public policy package once, and openshell-core API codegen references that same Rust type through extern_path.
• openshell-policy-schema owns generated public policy messages, bounded YAML and ProtoJSON codecs, canonical authored serialization, and intrinsic validation.
• openshell-policy owns public-to-internal lowering, base and effective projections, composition, and runtime-specific checks.
• Existing policy YAML shorthands and presence rules are either accepted by the thin compatibility layer or listed as intentional migration changes.
• The thin YAML layer covers every case in the authored-grammar inventory without defining a parallel policy object tree.
• YAML decoding enforces byte, depth, node, collection, scalar, alias, merge-key, tag, document-count, duplicate-key, and map-key limits.
• Public message presence distinguishes omitted filesystem_policy from an explicitly empty section.
• A presence-bearing public representation preserves the difference between omitted and explicitly empty MCP versions for YAML and direct SDK callers.
• Explicit null retains its current rejection behavior outside open middleware configuration.
• Required authored fields and the supported policy version fail with field-scoped validation errors instead of silently accepting protobuf defaults.
• Canonical serialization reconstructs documented conveniences such as tool, scalar glob matchers, compact single ports, and nested MCP options.
• Canonical serialization uses documented snake-case names, recursively sorted maps, and the existing default-omission rules.
• Public messages use buf.validate annotations for portable intrinsic constraints.
• The gateway validates every public policy regardless of whether it came from YAML, direct SDK construction, or gRPC.
• Rust, Go, Python, and TypeScript generate the public policy messages.
• Every SDK that advertises policy YAML support implements the thin YAML layer and passes the shared conformance corpus.
• The corpus covers valid policies, shorthand expansion, canonical output, malformed YAML, unknown fields, duplicates, null, presence, numeric bounds, internal-field rejection, normalization, and validation categories.
• Policy merge requests and policy-advisor messages no longer expose internal policy types.
• Provider-profile API messages stop importing internal endpoint types. The full provider-profile authored-schema refactor is tracked separately.
• The prover consumes a checked policy projection without serializing the internal proto to YAML and parsing it again.
• Documentation explains the public and internal policy models, canonical file form, compatibility period, migration command, SDK parsing, validation errors, and policy hash.
• Policy, SDK, API, storage, advisor, provider-composition, supervisor, and end-to-end tests pass.

Alternatives considered

Make the existing runtime protobuf public and authorable

This removes one type but exposes runtime authority, gives policy files normalized runtime field shapes, and couples the public contract to enforcement details. It is the wrong trust boundary.

Keep the #3334 Rust model as the permanent public schema

This gives Rust a good parser but leaves every other SDK to recreate the model. Public API types and policy files can still drift.

Store the public authored protobuf

This preserves the canonical authored message directly, but requires storage and hash migration without improving the current semantic round-trip guarantee. Keeping internal storage is the smaller first change. The option remains open if a future feature needs authoring provenance that normalization discards.

Require strict proto-shaped YAML and drop all shorthand

This makes each SDK parser smaller. It also creates avoidable churn across existing policy repositories and makes common rules harder to read. A small generic-value adapter is a reasonable cost for compatibility.

Preserve every YAML token

Comments, anchors, order, and formatting require a YAML syntax tree and edit-preserving writer. OpenShell does not preserve them today. They are outside the policy API contract.

Put all validation in Proto rules

Proto validation begins after decoding. It cannot make YAML safe, normalize a message, inspect provider catalogs, or validate runtime state. Encoding the full policy language in CEL would make security behavior harder to review.

Agent investigation

Current round trips are already canonical

parse_sandbox_policy parses the authored model and lowers it into the internal protobuf. serialize_sandbox_policy canonicalizes the protobuf, reconstructs the authored shape, and writes fresh YAML.

The tests check semantic equality after reparsing. They do not preserve source text. Current normalization includes:

• sorted map keys
• single-port compaction
• rule-name fallback
• MCP default materialization and version ordering
• scalar matcher reconstruction
• params["name"] reconstruction as MCP tool
• nested MCP and JSON-RPC configuration reconstruction
• omission of empty and default fields
• removal of internal provenance fields

Suggested Rust ownership

openshell-policy-schema
  generates openshell.policy.v1
  owns bounded YAML, ProtoJSON, canonical output, and intrinsic validation

openshell-core
  continues generating the existing public API and internal protocol messages
  maps openshell.policy.v1 to openshell-policy-schema with extern_path
  continues emitting the complete descriptor set

openshell-policy
  depends on public and internal policy messages
  owns lowering, projection, composition, merging, and runtime checks

openshell-sdk
  continues depending on openshell-core
  may depend directly on or re-export openshell-policy-schema for policy codecs

Creating an openshell-proto crate and moving the remaining generated messages out of openshell-core are not part of this issue. They can be done later as a dependency and ownership cleanup. That later change must preserve generated paths, re-exports, and descriptor coverage, but it does not block the public policy schema or API conversion work. It also would not remove the SDK's core dependency by itself because the SDK uses core for RPC errors, TCP_NODELAY helpers, and JWT expiry parsing.

Related work

#3333 and PR #3334 establish one dependency-light authored policy model in Rust. PR #3334 can land first. Its parser can remain the Rust compatibility path while the public proto and cross-language conformance suite are built.

The policy YAML and runtime-proto comparison gist shows why the current runtime proto should not become the file schema verbatim.

Checklist

• I've reviewed the current policy parser, protobufs, public API references, storage path, and related issue.
• This is a design proposal, not a request to start implementation.