NVIDIA / NVIDIA/NemoClaw

Measure selected configuration distributions

Open
#10,448 0 comments 0 reactions 1 assignee Claimed by @wscurran View on GitHub
area: observability area: sandbox area: security needs: design
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Outcome

Measure one-dimensional distributions of selected host and published-environment configuration without transmitting arbitrary configuration or constructing a per-environment profile.

This helps maintainers decide which platforms, runtime providers, GPU paths, and product options need compatibility work, tests, documentation, and support.

## Dependencies

- #10435 must define and receive approval for the telemetry privacy, service, schema, retention, and reporting contract.
- Extend the command-level post-commit registry observation mechanism from #10442. Do not create another observer or delivery path.
- The product and privacy owners must approve the closed signal and value allowlists.
- The common envelope from #10435 remains the only source of CPU architecture telemetry.

## OpenShell design alignment

OpenShell currently emits one `openshell_sandbox_create_event` after each sandbox-create attempt, including failed attempts. That event combines:

- Outcome.
- Whether the request asked for a GPU.
- Exact provider count.
- Whether a custom policy was present.
- Default, image, or undefined template source.
- A closed compute-driver category.

OpenShell does not send GPU verification, policy tier, host platform, web search, observability, or image ownership. It also does not send raw image or policy values.

This issue follows the OpenShell pattern for:

- Closed product-owned categories.
- Exact numeric counts.
- A shared anonymous product envelope with exact timestamps.
- Local mapping of arbitrary values to fallback categories.
- Best-effort delivery that cannot change product behavior.

NemoClaw intentionally measures successful persisted state, not create attempts. It emits one signal and value per event instead of combining configuration fields. Public reports therefore show marginal distributions, not joint environment profiles.

CPU architecture already appears in the common envelope. This issue must not emit a second architecture signal.

## Measurement

Emit `nemoclaw_configuration_observed` with:

| Field | Allowed values |
| --- | --- |
| `scope` | `host` or `sandbox` |
| `signal` | One approved signal for that scope |
| `value` | One value from that signal's allowlist |
| `count` | An exact positive integer for `sandbox` scope; absent for `host` scope |

Every event also uses the common envelope defined by #10435.

Initial signal allowlist:

| Scope | Signal | Allowed values |
| --- | --- | --- |
| `host` | `host_platform` | `linux`, `wsl`, `macos`, `other`, or `unknown` |
| `sandbox` | `compute_driver` | `docker`, `podman`, `kubernetes`, `other`, or `unknown` |
| `sandbox` | `gpu_state` | `not_configured`, `configured_unverified`, `verified`, `failed`, or `unknown` |
| `sandbox` | `web_search_enabled` | `true`, `false`, or `unknown` |
| `sandbox` | `observability_enabled` | `true`, `false`, or `unknown` |
| `sandbox` | `image_ownership` | `managed`, `custom`, or `unknown` |

Emit one host-platform event for each eligible observation batch.

For each sandbox signal, emit one event for each value with a nonzero count. The counts for each signal must equal the published environment count from #10442. If there are no published environments, emit no sandbox-configuration events.

Never combine two signals in one event. Never emit one row that describes one environment's complete configuration.

## Source of truth and eligibility

Use the complete post-commit sandbox registry snapshot.

Include only entries for which `isPublishedSandboxRegistration()` returns true. Exclude pending route reservations and unpublished recovery or staging state.

Treat every persisted field as untrusted input. The projector must use explicit type checks and allowlists. It must never spread, serialize, or log a registry entry.

If the registry cannot be read or projected as one complete snapshot, skip the complete configuration event family. Do not send partial counts.

## Host-platform mapping

Classify the host from `process.platform` and the existing narrow WSL detector:

- Linux with a positive WSL classification: `wsl`.
- Other Linux: `linux`.
- Darwin: `macos`.
- Another recognized platform: `other`.
- A classification failure: `unknown`.

Do not invoke the full readiness collector for telemetry. It performs Docker and GPU probes and reads more host state than this signal requires.

Do not transmit the WSL distribution, environment-variable values, kernel release, OS version, hostname, or other platform detail.

CPU architecture comes only from the common envelope. The configuration projector must not read or emit another architecture value.

## Compute-driver mapping

Use `SandboxEntry.openshellDriver`, which records the resolved compute plan rather than the gateway launcher.

Map values as follows:

- Exact `docker`: `docker`.
- Exact `podman`: `podman`.
- Exact `kubernetes`: `kubernetes`.
- Another present nonempty string: `other`.
- Missing, blank, or malformed value: `unknown`.

Do not normalize a missing or legacy value to Docker. Do not transmit the raw value or OpenShell version.

NemoClaw's active runtime-provider registry includes Podman, so exact `podman` maps to its named category. `vm` remains `other` until accepted product scope and schema review add it.

## GPU-state mapping

Use only `sandboxGpuEnabled` and the status fields from a structurally valid `sandboxGpuProof`:

| Recorded state | Telemetry value |
| --- | --- |
| `sandboxGpuEnabled === false` | `not_configured` |
| Enabled with `status === "verified"` and `cudaVerified === true` | `verified` |
| Enabled with `status === "failed"` and `cudaVerified === false` | `failed` |
| Enabled without a valid matching proof, including `unverified` or contradictory proof | `configured_unverified` |
| Missing or malformed enabled state | `unknown` |

An explicit disabled state wins over a stale proof.

Do not read or transmit host GPU detection, GPU mode, device selection, proof label, proof detail, proof timestamp, GPU model, or device identity.

This signal describes recorded configuration and proof state. It does not describe current GPU availability or health.

## Policy-tier deferral

Do not emit `policy_tier` in this issue's initial implementation. NemoClaw does not currently preserve the selected tier in final applied registry state; current normalization removes legacy policy-tier shadow fields.

Do not infer a tier from authored YAML, effective policy, preset names, custom policy contents, or live policy state.

Add this signal only after an accepted change durably preserves the selected tier in final applied state and #10435's closed signal and value schema is re-approved.

## Boolean-option mapping

For `web_search_enabled` and `observability_enabled`:

- Explicit `true` maps to `true`.
- Explicit `false` maps to `false`.
- Missing or malformed state maps to `unknown`.

Do not inspect or transmit a web-search provider, credential state, policy destination, observability endpoint, trace, or log.

## Image-ownership mapping

Classify only validated durable workload provenance:

- A valid `managed-image` workload receipt with no conflicting custom-image evidence maps to `managed`.
- A valid `legacy-dockerfile` workload receipt, or nonconflicting legacy `fromDockerfile` evidence, maps to `custom`.
- A native artifact, missing or malformed provenance, or conflicting provenance maps to `unknown`.

A native artifact is not an image and must not map to `managed` under this signal.

Do not transmit an image reference, tag, registry, Dockerfile path, digest, release, revision, cohort, platform, profile, or receipt field.

## Observation trigger

Use the shared command-level observer after a successful operation changes published environment membership or one measured configuration value.

Applicable completed operations include:

- Publishing or removing an environment.
- Completing onboarding or re-onboarding after final state and required cleanup.
- Completing a rebuild that changes measured configuration.
- Publishing a clone or restored environment.
- Completing recovery that publishes final measured state.

Observe only after the final registry commit and required cleanup. A multi-step operation produces at most one final configuration batch.

Do not observe after:

- A pending reservation, staging write, or intermediate registry mutation.
- A failed, cancelled, or rolled-back operation.
- A dry run or no-op operation.
- A read-only status, doctor, list, diagnostic, or health command.
- Agent start or stop.
- A model, provider, messaging, or policy mutation that does not change one of this issue's measured values.

Keep telemetry outside low-level registry mutation functions. Those functions also serve rollback, recovery, and tests.

## Privacy boundary

The client must never transmit:

- A registry row or arbitrary configuration key or value.
- Environment or sandbox names.
- Creation timestamps, reservation IDs, session IDs, transaction IDs, lifecycle IDs, or fingerprints.
- Hostnames, paths, mounts, ports, network destinations, or environment-variable values.
- OS distribution, OS version, kernel release, CPU model, hardware model, or serial number.
- GPU model, platform, device, mode, label, diagnostic, proof timestamp, or hardware inventory.
- Raw or unsupported driver names or OpenShell versions.
- Policy names, presets, custom policy contents, exclusions, receipts, or live policy results.
- Web-search provider or credential state.
- Observability endpoints, traces, logs, or collector configuration.
- Image references, tags, registries, Dockerfile paths, digests, releases, revisions, cohorts, profiles, or receipt contents.
- Model, provider, endpoint, credential, messaging, or user-content fields.
- Errors, stack traces, logs, or hashes of excluded values.

The service must not join configuration events into per-environment, per-install, per-user, or per-device profiles.

Exact timestamps and transport metadata can still correlate events in one batch. The privacy and service contract in #10435 must address that risk before implementation.

## Meaning and limitations

Reports must describe these values as:

- **Host-platform observations**, not unique hosts or operating-system inventory.
- **Configuration observations**, not unique or complete environment profiles.
- **Recorded GPU states**, not current GPU health.
- **Managed or custom image observations**, not image inventory.

Repeated eligible operations can count the same host and environment configuration again. Counts do not identify users, installations, hosts, or current running environments.

## Delivery and reporting

- Apply opt-out and CI or test suppression before host classification or registry access.
- Use the common best-effort delivery path from #10435.
- Short-lived CLI commands must use its bounded completion boundary. Do not rely only on an unjoined background queue.
- Make no retry and create no durable event outbox.
- Do not change onboarding, rebuild, recovery, policy, or lifecycle results when telemetry fails.
- Apply the public minimum-count rule from #10435.
- Publish each signal as a separate marginal distribution.
- Do not join configuration signals with environment, agent, model, provider, messaging, or install records.

## Acceptance criteria

- Only published registry entries contribute to sandbox counts.
- One eligible batch emits one host-platform event and no separate architecture event.
- Each sandbox signal's counts equal the published environment count.
- Events contain exactly one approved signal and one value.
- Compute drivers map to `docker`, `podman`, `kubernetes`, `other`, or `unknown` without exposing a raw value.
- Missing or legacy driver state never defaults to Docker.
- GPU state follows the defined precedence and never reads proof diagnostics or device details.
- No policy-tier event is emitted until final applied state preserves the selected tier and its closed schema is re-approved.
- Web search and observability use only explicit booleans or `unknown`.
- Image ownership uses only validated durable provenance, and native artifacts map to `unknown`.
- Pending, malformed, and conflicting state cannot produce a private raw value.
- A completed multi-step operation attempts at most one final configuration batch.
- Failed, cancelled, rolled-back, dry-run, no-op, intermediate, and read-only operations produce no configuration observation.
- Opt-out, CI, and test suppression prevent host classification, registry access, and network delivery.
- Delivery failure does not change the product result.
- Serialized events contain none of the prohibited fields.
- Public reporting follows the terminology, separation, and minimum-count contract from #10435.

## Test plan

Add deterministic tests for:

- A pure host-platform classifier with injected platform and WSL inputs.
- A pure registry-to-configuration-event projector.
- Zero, pending, published, legacy, malformed, and mixed registry rows.
- Exact per-signal count reconciliation.
- Docker, Podman, Kubernetes, and compute-driver fallback categories.
- GPU disabled, verified, failed, unverified, stale, malformed, and contradictory proof state.
- Absence of policy-tier events even when authored YAML, effective policy, or legacy shadow fields contain tier-like values.
- Explicit and missing web-search and observability booleans.
- Managed-image, legacy-Dockerfile, native-artifact, missing, malformed, and conflicting workload provenance.
- Event-schema rejection of multiple signals, dynamic keys, counts on host events, and missing counts on sandbox events.
- Absence of a separate architecture event.
- Private paths, hostnames, device IDs, image references, policy contents, endpoints, credential-shaped strings, control characters, and oversized values.
- Successful create, remove, re-onboard, rebuild, clone, restore, and recovery completion boundaries.
- Intermediate writes, failure, cancellation, rollback, dry-run, no-op, and read-only suppression.
- Opt-out, CI, test suppression, bounded CLI completion, and delivery failure.

Use synthetic registry entries and a local contract server. No live NVIDIA service, OpenShell gateway, sandbox, GPU, or external API is required.

## Not included

- A second CPU architecture measurement.
- Unique installation, host, user, or environment identity.
- Complete or joint per-environment configuration profiles.
- Arbitrary configuration inventory.
- Policy-tier telemetry until final applied state durably preserves the selected tier and its closed schema is re-approved.
- Detailed OS, CPU, GPU, driver, policy, image, network, or observability inventory.
- Live sandbox, GPU, policy, service, or workload health.
- Model, provider, inference, messaging, or agent activity measurement.
- Continuous polling or timers.
- A general telemetry API for plugins.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.