feat(k8s): support ephemeral emptyDir-backed sandbox workspaces as an alternative to the default PVC

User Story

As a platform operator running OpenShell on a managed Kubernetes cluster (ROSA, EKS, AKS, GKE, Gardener) or on a short-lived CI cluster, I want to opt sandboxes into an ephemeral, emptyDir-backed workspace instead of the per-sandbox PVC, so that I can run OpenShell in environments where PVC provisioning is unavailable, unreliable, expensive, or disallowed by cluster policy — without giving up sandbox lifecycle correctness.

Problem Statement

The Kubernetes compute driver unconditionally injects a workspace PVC into every sandbox pod via volumeClaimTemplates on the agents.x-k8s.io/Sandbox CR (crates/openshell-driver-kubernetes/src/driver.rs:2041-2176, constants at :413-431). After PR #2088 removed SandboxTemplate.volume_claim_templates from the public API, there is no supported way to run a sandbox pod without a PVC. The only knobs available today are workspace_default_storage_size (#1436) and workspace_storage_class (#2442, PR #2463) — both change which PVC gets created, not whether one is created.

An emergent workaround exists (has_explicit_sandbox_data_mount disables default PVC injection when the operator supplies any driver_config subPath mount under /sandbox), but it is undocumented, requires supplying an unrelated PVC just to trigger the branch, and does not deliver emptyDir semantics. It is an implementation side-effect, not a contract.

The agents.x-k8s.io/Sandbox CRD does not require the workspace to be a PVC — volumeClaimTemplates is optional in api/v1beta1. This is entirely an OpenShell-side rendering assumption, so the change is scoped to this repo.

Impact / Why This Matters

Without an ephemeral option, users of the Kubernetes driver today must:

• Provide a StorageClass in every target cluster, even when the sandbox does not need pod-reschedule survival. On clusters without a default StorageClass and without workspace_storage_class set, the workspace PVC stays Pending and the sandbox never starts. This blocks OpenShell adoption on short-lived CI clusters, edge clusters, restricted managed clusters, and Gardener-style shoots where dynamic provisioning is not always installed.
• Pay the provisioning cost and lifecycle risk of a PVC per sandbox, even for short-lived agent tasks that never restart. The PVC + init-container path adds first-start latency (called out as a stopgap in the code comment at driver.rs:399-411) and, as documented in #1879, leaves orphaned PVCs holding written user data on teardown paths that skip the finalizer (default agent-sandbox shutdownPolicy: Retain).
• Give up the fail-safe property emptyDir provides — kubelet-guaranteed reclaim, no external object to orphan, no dependence on cleanup being correct. #1879 already identifies this as a valid workspace model, but scopes it to warm-pooled sandboxes only, leaving cold-path sandboxes without the option.
• Rely on an undocumented driver-config side-effect (has_explicit_sandbox_data_mount) to skip the PVC — not a stable contract and requires supplying a foreign PVC just to disable default injection.

The maintainer-authored design for the current PVC path (#743) explicitly anticipated this: "Opt-out mechanism: some users may want ephemeral sandboxes. Consider --no-persist flag or server env var. Can be a follow-up." No follow-up has been filed.

Proposed Design

Add a workspace backend selector to the Kubernetes driver configuration, mirroring the shape of workspace_storage_class (#2442) exactly.

Gateway-wide default (crates/openshell-driver-kubernetes/src/config.rs):

#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum WorkspaceBackend {
    /// Per-sandbox PVC via `volumeClaimTemplates` on the Sandbox CR.
    /// Survives pod rescheduling. Requires a StorageClass.
    #[default]
    Pvc,
    /// `emptyDir` volume mounted at `/sandbox`. No PVC created.
    /// Ephemeral — deleted with the pod.
    Ephemeral,
}

pub struct KubernetesComputeConfig {
    // ...existing fields...
    pub workspace_backend: WorkspaceBackend,      // new
    pub workspace_default_storage_size: String,   // ignored when Ephemeral
    pub workspace_storage_class: String,          // ignored when Ephemeral
    // ...
}

• Same driver-config → gateway TOML → Helm value fan-out that the existing storage-class field uses (server.workspaceBackend).
• Default is Pvc, preserving current behavior byte-for-byte.

Per-sandbox override — no new proto or CLI surface required. The existing SandboxTemplate.driver_config envelope (proto/openshell.proto:862) is already the driver-keyed opaque config the gateway forwards to the compute driver, and the CLI already exposes it as --driver-config-json. The only change is: the Kubernetes driver reads workspace_backend from the per-request struct as it already does for other overridable fields.

# Ephemeral sandbox on a gateway defaulted to pvc:
openshell sandbox create --name ephemeral-task \
  --driver-config-json '{"kubernetes":{"workspace_backend":"ephemeral"}}' \
  -- claude

# Persistent sandbox on a gateway defaulted to ephemeral:
openshell sandbox create --name durable-task \
  --driver-config-json '{"kubernetes":{"workspace_backend":"pvc","workspace_default_storage_size":"10Gi"}}' \
  -- claude

Rendering (crates/openshell-driver-kubernetes/src/driver.rs, apply_workspace_persistence):

• Pvc → existing path unchanged.
• Ephemeral → skip volumeClaimTemplates entirely; add a pod-spec emptyDir volume named workspace at /sandbox (WORKSPACE_MOUNT_PATH); the workspace-init init container still seeds /sandbox from the image on every pod start (sentinel unnecessary — volume is always empty at pod start).

Observable behavior

• sandbox create in Ephemeral mode succeeds on a cluster with no StorageClass installed.
• sandbox stop + sandbox start on an ephemeral sandbox starts with a fresh /sandbox — documented as the ephemeral contract, complementary to the PVC-backed lifecycle from #2652.
• Pod eviction / rescheduling on an ephemeral sandbox loses /sandbox state (documented). PVC-backed sandboxes retain today's behavior unchanged.
• kubectl get pvc -n <ns> shows zero workspace PVCs for ephemeral sandboxes.

Acceptance Criteria

• driver_config.kubernetes.workspace_backend accepts "pvc" (default) and "ephemeral"; unknown values rejected at config parse.
• With workspace_backend = "ephemeral", the rendered pod spec contains an emptyDir volume at /sandbox and the Sandbox CR contains no volumeClaimTemplates entry.
• With workspace_backend = "ephemeral", sandbox creation succeeds on a cluster that has no default StorageClass and no explicit workspace_storage_class.
• The workspace-init init container seeds /sandbox from the image on every pod start in ephemeral mode.
• Default (unset) behavior is byte-for-byte identical to today's PVC path — asserted by a render-diff test.
• Per-sandbox override via --driver-config-json '{"kubernetes":{"workspace_backend":"..."}}' works and takes precedence over the gateway default.
• Combining workspace_backend = "ephemeral" with workspace_default_storage_size or workspace_storage_class fails validation with a clear error.
• Helm value server.workspaceBackend and the gateway TOML field render correctly through the deployment path and are documented in docs/reference/gateway-config.mdx and docs/reference/sandbox-compute-drivers.mdx.
• Kubernetes e2e: create an ephemeral sandbox, write a marker to /sandbox, delete the pod, verify (a) no PVC is left behind and (b) a re-created sandbox with the same identity starts with a fresh /sandbox.

Alternatives Considered

• Do nothing; document the has_explicit_sandbox_data_mount side-effect. Keeps the emergent workaround as the answer. Requires operators to supply an unrelated PVC just to disable default injection, does not deliver emptyDir semantics, and remains contract-less.
• Generic Ephemeral Volumes (ephemeral.volumeClaimTemplate on the pod spec). Still requires a StorageClass; does not solve the "cluster has no provisioner" case. Complementary; could be added later as a third workspace_backend variant.
• tmpfs for /sandbox. Bounded by pod memory limits and interacts poorly with the image-seed path (image content can be arbitrarily large). Not a fit for the default case.
• Wait for the container-snapshotting replacement referenced by driver.rs:399-411. There is no tracking issue for that work; the ephemeral option is complementary and useful regardless of when snapshotting lands.
• Reintroduce SandboxTemplate.volume_claim_templates. Explicitly reverted by PR #2088. This proposal instead follows the current direction (customize through driver_config, not by opting the whole PVC surface out via raw CRs).

Agent Investigation

• Kubernetes driver injects the workspace PVC unconditionally: crates/openshell-driver-kubernetes/src/driver.rs:413-431 (constants), :2041-2139 (apply_workspace_persistence), :2150-2176 (default_workspace_volume_claim_templates).
• Emergent bypass via subPath mounts: has_explicit_sandbox_data_mount disables default injection for any driver-config mount at-or-under /sandbox (noted by @elezar in PR #2034 review).
• PR #2088 removed SandboxTemplate.volume_claim_templates from the public API, closing the previous raw-CR escape hatch and consolidating storage configuration behind driver_config.kubernetes.
• Existing driver-keyed override pipe: SandboxTemplate.driver_config (proto/openshell.proto:862) is already the opaque per-request envelope the gateway forwards to the compute driver — no proto change required for the per-sandbox override.
• CLI already exposes the override generically: --driver-config-json on sandbox create (crates/openshell-cli/src/main.rs:1386-1392, parser in crates/openshell-cli/src/run.rs:262) — no CLI change required.
• #743 (design spike for the current PVC path) explicitly deferred the ephemeral opt-out as a follow-up that never materialized.
• #1879 (warm-pooled sandboxes) proposes emptyDir as one of two workspace models — but only on the warm path, leaving cold-path sandboxes without the option.
• Precedent for the config-knob shape and rollout: #2442 → PR #2463 (workspace_storage_class) — identical driver/TOML/Helm/docs fan-out.
• Managed-cluster context: #899 (restricted SCC on OpenShift) confirms the managed-Kubernetes audience is present upstream; discussion #2626 (edge devices) is an adjacent audience for the same "PVC undesirable" case.
• The agents.x-k8s.io/Sandbox CRD (kubernetes-sigs/agent-sandbox, api/v1beta1) does not require volumeClaimTemplates — the field is optional, so this proposal needs no operator or CRD change.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

Start with crates/openshell-driver-kubernetes/src/config.rs and driver.rs, especially apply_workspace_persistence and the workspace constants, then trace the existing workspace_storage_class configuration through the gateway TOML and Helm deployment. Review the CLI driver-config entry points and the two documentation files named in the acceptance criteria. Done means the PVC default remains unchanged, ephemeral rendering and overrides are validated by render tests, and the Kubernetes e2e verifies fresh workspace state without a PVC.