feat(helm)!: replace mirrored gateway settings with YAML-to-TOML configuration

User Story

As a Kubernetes operator, I want to configure the gateway through one structured Helm value, so that I can use the complete gateway configuration contract without waiting for the chart to expose every field separately.

Problem Statement

The OpenShell chart currently mirrors gateway configuration fields as individual Helm values and reconstructs gateway.toml in templates/gateway-config.yaml. The template currently reads approximately 65 distinct Helm value paths spanning gateway settings, Kubernetes driver settings, credential drivers, TLS, OIDC, OTLP, rate limiting, images, namespaces, and proxy behavior.

This duplicates the gateway's TOML schema in the chart. Adding, renaming, moving, or removing a gateway setting requires coordinated changes to:

• the gateway TOML parser and reference documentation;
• values.yaml and generated chart documentation;
• the ConfigMap template;
• chart unit tests and CI overlays; and
• upgrade guidance for both the TOML and Helm representations.

Operators also cannot use a newly supported gateway TOML field until the chart adds a dedicated value and rendering branch.

Impact / Why This Matters

The Helm chart has become a second field-by-field application configuration API rather than a Kubernetes packaging layer. The duplicated contract can drift from gateway.toml, delays access to new settings, and makes the coordinated 0.1.0 stabilization work larger than necessary.

The current workaround is to add another chart value and template branch for each gateway field or fork the rendered ConfigMap. That workaround is insufficient because it repeats for every schema change and requires operators to learn a separate Helm-specific location for every gateway setting.

Proposed Design

Follow Grafana's grafana.ini pattern: replace individually mirrored gateway application settings with one top-level structured YAML map, then convert that map into the mounted TOML file.

Illustrative values shape:

gatewayConfig:
  openshell:
    version: 1

  openshell.gateway:
    name: '{{ include "openshell.fullname" . }}'
    bind_address: '0.0.0.0:{{ .Values.service.port }}'
    log_level: info

  openshell.gateway.oidc:
    issuer: https://issuer.example.com
    audience: openshell-cli

  openshell.drivers.kubernetes:
    workspace_mode: shared
    gateway_id: '{{ .Release.Name }}'

This renders as:

[openshell]
version = 1

[openshell.gateway]
name = "release-openshell"
bind_address = "0.0.0.0:8080"
log_level = "info"

[openshell.gateway.oidc]
issuer = "https://issuer.example.com"
audience = "openshell-cli"

[openshell.drivers.kubernetes]
workspace_mode = "shared"
gateway_id = "release"

Externally observable behavior:

• Top-level gatewayConfig keys represent TOML table names; their values contain the fields for that table.
• YAML scalar types remain typed when rendered: strings are TOML-quoted and escaped, while booleans and numbers remain booleans and numbers.
• YAML lists render as TOML arrays. The representation and rendering behavior for TOML arrays of tables and inline tables is documented and tested.
• String values may use Helm template expressions and are evaluated with tpl, matching Grafana's handling of string values.
• The rendered TOML becomes the gateway.toml key in the gateway ConfigMap and is mounted at /etc/openshell/gateway.toml.
• The chart supplies a working default gatewayConfig map for a standard Kubernetes installation.
• The conversion is generic: it serializes the supplied tables and values without maintaining a gateway-field allowlist.
• Values that configure Kubernetes resources remain normal Helm values, including workload controller, Service ports, probes, RBAC, ServiceAccounts, volumes, Secrets, certificate resources, routes, and network policies.
• Secret material remains outside the ConfigMap. Database URLs, credentials, private keys, and similar data continue to use Kubernetes Secrets and supported environment/file expansion.
• Chart-owned values that also affect gateway behavior have one documented ownership model. The default map may reference infrastructure values with tpl, or the chart may pass a required runtime override through existing CLI/environment precedence; the chart must not expose a second independent gateway-setting knob.

Grafana's current chart defines one grafana.ini map and converts its top-level values and section maps into INI, applying tpl to strings:

• Grafana chart values
• Grafana config rendering helper

OpenShell should apply the same model to TOML while supporting the TOML constructs used by gateway and extension configuration.

Required Legacy-Value Removal

This is a breaking contract cleanup, not an additive escape hatch. Implementation must inventory every value referenced by deploy/helm/openshell/templates/gateway-config.yaml, remove the field-by-field TOML construction, and classify each current value by ownership.

The finished gateway-config.yaml must be a generic serializer. It may consume .Values.gatewayConfig and the Helm context needed to evaluate templated string values, but it must not contain gateway-field-specific reads, conditionals, defaults, validation, or table construction. In particular, all of the approximately 65 current .Values references must be removed from that template. Gateway schema defaults and validation belong to the gateway parser.

Remove from the Helm contract and express under gatewayConfig

Remove values whose only purpose is to populate gateway.toml, including these existing families:

• server.name, server.logLevel, server.otlp.*, server.auth.*, server.policyValidationFailureMode, server.grpcRateLimit.*, and server.enableLoopbackServiceHttp;
• gateway OIDC fields such as issuer, audience, JWKS TTL, role claims, and scope claims;
• sandbox and Kubernetes-driver behavior such as the default image, image pull policy, workspace storage defaults, default RuntimeClass, AppArmor profile, user namespaces, operator namespace selector/file, and sandbox JWT gateway ID and TTL values;
• upstream proxy URL, no-proxy, authentication behavior, and hostname-resolution behavior;
• Vault connection/authentication fields and Kubernetes Secrets driver settings that do not create, mount, or authorize Kubernetes resources; and
• any other current server.*, supervisor.*, upstreamProxy.*, service.*, networkPolicy.*, certManager.*, or pkiInitJob.* value that is referenced solely to emit a TOML key.

Defaults for these settings move into the default gatewayConfig map only when the chart needs a Kubernetes-specific default. Otherwise, omit them and let the gateway configuration schema provide its own default. Do not keep deprecated aliases or shadow values that rebuild the old field-by-field API.

Keep only Kubernetes packaging inputs

Retain separate Helm values only where Helm uses them to create or modify Kubernetes resources. The implementation review must cover at least:

• workload kind, replicas, gateway/supervisor container images, probes, Service ports, routes, pod settings, and rollout behavior;
• ServiceAccounts, RBAC, sandbox namespace lifecycle, NetworkPolicy resources, and image-pull Secret permissions;
• TLS and JWT Secret names/mounts, cert-manager resources, and the PKI initialization Job;
• database Secret/volume wiring, OIDC CA ConfigMap mounting, SPIFFE socket mounting, host aliases, and credential-driver Secret/RBAC resources; and
• supervisor delivery/topology settings and driver modes only where they materially alter rendered Kubernetes resources or permissions.

A retained packaging value must have one owner. If the corresponding gateway setting is required, the default gatewayConfig may derive it with tpl, or the chart may pass it through the existing CLI/environment override layer. The chart must not expose both a packaging value and an independently configurable duplicate under the legacy gateway path. Values that currently mix both responsibilities should be renamed or split so their Kubernetes purpose is explicit.

Migration instructions

Document an old-to-new mapping for every removed value, update all chart overlays and examples, and replace field-specific ConfigMap tests with generic serializer tests plus real-parser validation. Include explicit examples for OIDC, OTLP, Kubernetes driver defaults, credential drivers, proxy configuration, TLS, and rate limiting.

Acceptance Criteria

• The chart exposes one documented gatewayConfig YAML map as the gateway application configuration contract.
• Top-level map keys render as TOML tables in gateway.toml.
• Strings, booleans, integers, floats, arrays, inline tables, and arrays of tables have documented deterministic YAML-to-TOML conversion behavior.
• String values support Helm tpl expressions using release metadata and chart infrastructure values.
• TOML strings and keys are quoted and escaped safely; values cannot inject additional TOML entries through malformed quoting.
• The default map produces a valid, bootable Kubernetes gateway configuration.
• A custom gateway field unknown to the Helm chart is serialized and reaches the gateway without a chart template change.
• The converter does not maintain a gateway-field allowlist or duplicate gateway defaults and validation.
• Individually mirrored gateway application fields are removed from values.yaml, chart schema and documentation, templates, examples, and tests; no compatibility aliases remain.
• Kubernetes-resource values remain separate and are clearly distinguished from gateway application configuration.
• The migration guide provides an old-to-new mapping for every removed value and calls out the breaking upgrade.
• Secret-bearing gateway configuration remains sourced from Kubernetes Secrets or supported environment/file expansion and is not rendered into the ConfigMap.
• The gateway workload checksum changes when the rendered TOML changes, triggering a rollout.
• Chart tests cover default rendering, unknown fields, every supported TOML value shape, tpl evaluation, escaping, omitted-versus-empty values, secret handling, and rollout checksum changes.
• The rendered default and representative advanced configurations are parsed by the real gateway configuration parser in CI.
• Chart documentation, CI overlays, docs/reference/gateway-config.mdx, and the 0.1.0 migration guide are updated.
• Adding a non-secret gateway TOML field in the future does not require changing the Helm chart.

Alternatives Considered

Keep one Helm value per gateway field

This provides field-specific Helm documentation and render-time validation, but it preserves the duplicated contract and requires chart work for every gateway schema change.

Accept a raw TOML string

A raw string keeps Helm out of serialization entirely, but it is less natural for normal values.yaml, --set, and values-composition workflows. A structured YAML map follows Grafana's established chart pattern and gives operators typed values while retaining one application-config boundary.

Recursively infer TOML tables from arbitrary nested YAML

This can make simple configuration shorter, but dotted TOML table names, inline tables, and arrays of tables can become ambiguous. An explicit, documented mapping from top-level section names to TOML tables makes the contract easier to reason about. Nested values should be supported only where their TOML representation is unambiguous.

Merge a user-supplied configuration fragment into generated configuration

This creates ambiguous precedence and difficult structural merge behavior for tables and arrays of tables. One complete gatewayConfig map gives operators a deterministic result.

Reference only an existing ConfigMap

An existing-ConfigMap mode could be added later for externally managed configuration, but it does not simplify the default values-based chart contract.

Agent Investigation

• deploy/helm/openshell/templates/gateway-config.yaml currently reconstructs the full TOML file and references approximately 65 unique .Values paths.
• deploy/helm/openshell/tests/gateway_config_test.yaml contains extensive field-by-field assertions against generated TOML, demonstrating the maintenance cost of the mirrored schema.
• Gateway configuration precedence already supports this boundary: CLI flags and OPENSHELL_* environment variables override the TOML file, while the gateway parser owns field validation and defaults.
• #2792 addresses inconsistencies inside the gateway TOML schema. This issue is complementary: it removes the need to repeat the stabilized schema as individual Helm values.
• Grafana's actively maintained community chart exposes one grafana.ini map and renders it through a generic ConfigMap helper. OpenShell can use the same separation and typed values while emitting TOML.

Related: #2565, #2792.