NVIDIA / NVIDIA/NemoClaw

Measure published environments and configured agent runtimes

Open
#10,442 1 comment 0 reactions 1 assignee Claimed by @wscurran View on GitHub
area: observability area: security needs: design
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Outcome

Measure how many published NemoClaw environments exist and how many agent runtimes are configured across them, without identifying an environment, inspecting a running process, or transmitting registry records.

This helps us understand the environments users manage and the agent versions we need to support.

## Dependencies

- #10435 accepted the telemetry privacy, service, and reporting contract.
- #10904 accepted multi-sandbox, multi-agent YAML as the future desired-state contract.
- Production delivery remains disabled until Data Platform provisioning and UAT complete.
- YAML lifecycle wiring waits for a separately accepted mutating #10904 child that persists final applied agent state.

## OpenShell design alignment

Follow the OpenShell client pattern established by #9540:

- Send exact counts rather than count buckets.
- Use the common anonymous client identifier and exact event timestamps.
- Project local state into fixed event schemas.
- Use closed, product-owned categories instead of arbitrary values.
- Attempt delivery once after a completed operation.
- Do not retry, queue, or change the product result when telemetry fails.
- Respect the common opt-out and automatic CI and test suppression.

NemoClaw must project one stable snapshot of durable applied state bound to published registry rows. It must never project authored YAML or serialize or transmit a registry entry.

## Measurements

| Event | Fields | Meaning |
| --- | --- | --- |
| `nemoclaw_sandbox_count_observed` | `count` | Exact number of published environment registrations |
| `nemoclaw_agent_runtime_observed` | `agent_runtime`, `count` | Exact number of configured agents using each approved runtime |
| `nemoclaw_managed_agent_version_observed` | `agent_runtime`, `managed_agent_version`, `count` | Exact number of configured agents using each approved NemoClaw-managed version |

Every event also uses the common envelope defined by #10435.

The total environment event may report zero. Runtime and managed-version events are emitted only for categories with a nonzero count.

## Environment eligibility

Use `isPublishedSandboxRegistration()` in `src/lib/state/registry/route-reservation.ts` as the authoritative eligibility rule.

Include a registry row only when:

```text
pendingRouteReservation !== true
```

Exclude:

- Pending route reservations.
- Staging and recovery checkpoints that have not been published.
- Rows already removed by a completed lifecycle operation.
- Source checkouts or other state not represented by a published registry row.

A published registry row is a configured environment observation. It does not prove that a sandbox or agent process is currently running or healthy.

## Agent runtime categories

Map every durable configured agent through these closed categories:

- `openclaw`
- `hermes`
- `langchain-deepagents-code`
- `other`

For YAML-managed state, each bound `spec.sandboxes[].agents[]` entry counts once. Repeated runtime types count repeatedly. For a legacy published row, only its bounded registered `agent` counts; do not decode opaque extra-agent transports. Missing, legacy, custom, and unrecognized values map to `other`. Never send the original value.

The sum of all emitted runtime-category counts must equal the number of projected configured-agent entries. Agent totals are independent from, and may exceed, the published environment count.

This measurement describes durable configured agents. It does not count processes, dynamically created subagents, sessions, or agent activity.

## Managed agent version categories

Report a managed agent version only when all of these conditions hold:

- The owning sandbox is a published environment.
- The configured agent and version are bound to final applied state or bounded legacy registration state.
- Product-owned managed workload provenance applies to that exact agent.
- The runtime is an approved runtime category.
- The version matches a checked-in, product-owned version key.

Map all other managed-image version values to `other`. This includes missing, malformed, legacy, and unrecognized versions.

Do not derive or report a managed version for:

- Custom images.
- Legacy Dockerfile images.
- Native or externally managed artifacts.
- Arbitrary image references or tags.

Never transmit a raw custom version, image reference, source revision, source cohort, Dockerfile path, or image tag.

## Observation trigger

After a successful lifecycle operation changes published registry membership, the durable configured-agent set, or managed-version provenance, project the complete post-commit applied state and attempt one observation batch.

Applicable completed operations include:

- Publishing or creating an environment.
- Removing an environment.
- Restoring an environment during successful recovery.
- Publishing a cloned or restored environment.
- Completing a rebuild that changes the configured runtime or managed version.

Observe only after the top-level operation reaches verified `succeeded` and all final state is committed and cleaned up. A multi-step or multi-sandbox operation produces at most one complete batch and no intermediate observations.

Do not observe after:

- A pending reservation is written.
- A staging or rollback checkpoint is written.
- Export, validation, planning, or another read-only command.
- A no-op or a registry write that does not change an eligible measurement.
- A lifecycle result of `failed-before-effects`, `partially-applied`, `recovery-required`, or `inconclusive`.
- A cancelled or rolled-back operation.

Keep telemetry outside the low-level registry mutation functions. Those functions are shared by normal operations, rollback, recovery, and tests; placing network behavior there could expose intermediate state or produce duplicate observations.

If the applied state cannot be read or projected as one complete snapshot, skip the entire batch. Do not send partial counts.

Do not add YAML lifecycle telemetry to the read-only export slice. A later accepted mutating YAML child must own the post-success hook and the durable per-agent provenance it reads.

## Privacy boundary

The event projector must use an explicit field allowlist. It must not transmit:

- Environment names.
- Creation timestamps.
- Reservation, session, transaction, or lifecycle identifiers.
- Paths, mounts, image references, digests, or source revisions.
- Models, inference providers, API endpoints, or credentials.
- Messaging configuration.
- Policies, policy names, or custom policy contents.
- GPU devices or detailed host configuration.
- Registry rows or arbitrary keys and values.

No geography is derived from environment state. The service may classify the event request’s source IP to country only if the accepted #10435 contract permits it.

## Delivery behavior

- Apply the common default-on opt-out behavior before reading registry state.
- Disable delivery in CI and automated tests.
- Make at most one bounded delivery attempt per completed operation.
- Do not add a retry queue or durable outbox.
- Do not change lifecycle output, exit status, cleanup, or rollback behavior when observation fails.
- Do not include an environment identifier in the common envelope or event fields.

Repeated observations can include the same environment. Downstream reports must describe these as environment observations, configured agent runtimes, and configured managed agent versions—not unique environments, active agents, or the current installed base.

## Acceptance criteria

- Published environments are counted exactly once; pending rows are excluded and zero is valid.
- Every durable configured agent is counted through a closed runtime category; agent totals may differ from environment totals and raw values never appear.
- Managed versions are counted only from product-owned provenance; custom or unverified values never appear verbatim.
- One complete batch is attempted only after a top-level operation reaches verified success and commits all final state; read-only, no-op, intermediate, failed, partial, recovery, inconclusive, and rollback states emit nothing.
- An unreadable or invalid snapshot produces no partial batch.
- Opt-out, CI, and test suppression prevent applied-state reads and network delivery.
- One five-second delivery failure does not change the lifecycle result and is never retried or queued.
- Local tests prove the closed schemas, excluded data, multi-agent counts, suppression, and receiver behavior while production delivery remains disabled.

## Test plan

Add deterministic tests for:

- A pure applied-state-to-event projector using synthetic snapshots.
- Zero, pending, published, legacy, custom, mixed-runtime, and repeated-runtime multi-agent snapshots.
- Managed-image and non-managed workload receipts.
- Exact count reconciliation.
- Serialization allowlists and prohibited-field absence.
- Successful single- and multi-sandbox completion boundaries after final state is committed.
- Read-only, no-op, failed, partial, recovery, inconclusive, rollback, and intermediate states.
- Opt-out, CI, test suppression, and delivery failure.

Use a local contract server for delivery tests. No live NVIDIA service, sandbox, or external API is required.

## Not included

- Detecting running processes, agent health, sessions, or internal subagents.
- Model or inference-provider measurement.
- Messaging measurement.
- Configuration measurement.
- Unique environment identity.
- Continuous inventory polling or timers.
- Raw custom runtime or version values.
- Runtime success, failure, or activity events.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.