NVIDIA / NVIDIA/NemoClaw

[Design] Replace the Voice Gateway with a runtime-neutral Agent Session API

Open
#11,204 0 comments 0 reactions 0 assignees View on GitHub
area: architecture needs: design
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Parent

- #8275

## Decision status

Pending maintainer decision. This issue does not authorize implementation.

An `Accept` decision must state the reason, placement, accountable maintainer, and validation plan.

## Proposal

Image

Replace the experimental Voice Gateway API with a private, runtime-neutral NemoClaw Agent Session API.

The ownership boundary is:

- VoiceClaw owns the user interaction session, task decomposition, task IDs, scheduling, and cross-task supervision.
- NemoClaw owns authentication, trusted agent binding, one multi-turn agent session, event delivery, cancellation, and runtime adaptation.
- One VoiceClaw subtask maps to one NemoClaw Agent Session.
- One NemoClaw Agent Session maps to one downstream ACP session.
- NemoClaw does not relate or supervise multiple VoiceClaw subtasks.

A deployment may initially limit active Agent Sessions to one. The API and state model must not use a process-wide singleton.

## Runtime-facing contract

Base path:

```text
/v1alpha1/agent-sessions
```

| Operation | Purpose |
|---|---|
| `POST /agent-sessions` | Create one multi-turn Agent Session |
| `GET /agent-sessions/{sessionId}` | Read session state |
| `POST /agent-sessions/{sessionId}/turns` | Commit one text turn |
| `GET /agent-sessions/{sessionId}/events?after={sequence}` | Receive ordered SSE events or replay retained events |
| `POST /agent-sessions/{sessionId}/turns/{turnId}/cancel` | Cancel the active turn |
| `DELETE /agent-sessions/{sessionId}` | Close the Agent Session |

Example creation request:

```json
{
"clientSessionId": "voiceclaw-subtask-42",
"interactionProfile": "task-steering-v1"
}
```

Example turn request:

```json
{
"clientTurnId": "turn-7",
"text": "Check whether the deployment is healthy."
}
```

Commands return after admission. Results use SSE events:

```text
session.created
turn.accepted
turn.started
response.text.delta
progress.updated
turn.completed
turn.failed
turn.cancelled
session.closed
session.failed
session.expired
```

Each event contains a session ID, turn ID when applicable, event ID, and monotonic per-session sequence number.

`turn.completed` ends one agent turn. It returns the Agent Session to `idle`; it does not declare the VoiceClaw subtask complete. VoiceClaw decides whether to send another turn or close the Agent Session.

Only one turn may run at a time in one Agent Session. Multiple independent Agent Sessions may run when configured capacity permits. A client disconnect does not cancel agent work.

## Internal ACP adapter

The public API must not expose ACP. NemoClaw uses this internal boundary:

```ts
interface AgentSessionAdapter {
initialize(): Promise;
createSession(): Promise;
prompt(
session: RuntimeSession,
text: string,
onUpdate: (update: RuntimeUpdate) => void,
): Promise;
cancel(session: RuntimeSession): Promise;
close(session: RuntimeSession): Promise;
}
```

The first proposed implementation is `AcpAgentSessionAdapter`:

| Adapter operation | ACP operation |
|---|---|
| `initialize` | `initialize` and capability validation |
| `createSession` | `session/new` |
| `prompt` | `session/prompt` |
| `onUpdate` | receive `session/update` |
| `cancel` | `session/cancel` |
| `close` | `session/close` when qualified, then terminate the transport |

For OpenClaw, NemoClaw would launch a qualified `openclaw acp` server and exchange ACP JSON-RPC over a duplex stream. The accepted implementation must pin and qualify the complete NemoClaw, OpenShell, OpenClaw, and ACP tuple.

Raw ACP frames, native session IDs, tool arguments, reasoning, terminal access, and permission requests must not cross the runtime-facing API.

## Security and lifecycle

- Trusted NemoClaw configuration selects the agent, sandbox, working directory, ACP command, MCP configuration, and upstream endpoint.
- VoiceClaw cannot provide or override those values.
- A deployment credential may create Agent Sessions.
- NemoClaw returns a short-lived session grant, stores only its hash, and revokes it on close or expiry.
- Agent and OpenShell credentials remain in their owning trust boundaries and never reach VoiceClaw.
- NemoClaw policy resolves permission requests. VoiceClaw cannot approve them in V1.
- Ambiguous transport failure must not replay a prompt automatically.
- Close or expiry revokes the grant and terminates the owned ACP transport. It preserves the agent sandbox and unrelated sessions.
- Restart recovery remains unsupported until durable event replay and ACP resume pass qualification.

## Relationship to existing work

- If accepted, amend #8275 so VoiceClaw retains supervisor-session ownership.
- Supersede the runtime contract proposed in #9646.
- Re-scope #8403 from VoiceClaw deployment identity to Agent Session admission credentials.
- Replace the context-sharing behavior from #9411 with one stable Agent Session per VoiceClaw subtask.
- #10947 is related ACP transport work for Hermes. It does not establish OpenClaw ACP support.

## Non-goals

- A2A
- VoiceClaw task scheduling or cross-task supervision
- Audio, ASR, TTS, WebRTC, RTVI, or UI protocols
- Concurrent turns within one Agent Session
- Client-selected agents, sandboxes, tools, or upstream endpoints
- Qualification of every ACP agent
- Restart recovery in V1

## Acceptance criteria

- [ ] Maintainers record `Accept`, `Request changes`, `Defer`, or `Decline`.
- [ ] An `Accept` decision names the reason, placement, accountable maintainer, and validation plan.
- [ ] The ownership boundary and one-subtask-to-one-Agent-Session mapping are accepted.
- [ ] The HTTP command and SSE event contract is accepted.
- [ ] Identity, idempotency, ordering, replay, cancellation, detach, expiry, and terminal semantics are assigned.
- [ ] The internal adapter and ACP mapping are accepted.
- [ ] The supported runtime tuple and duplex transport are selected.
- [ ] Credential custody, authorization, cleanup, and negative security evidence are defined.
- [ ] Disposition is recorded for #8275, #9646, #8403, and #9411.
- [ ] Implementation issues are created only after an `Accept` decision.

## Category

Feature

Contributor guide

Open the contributing guide

Research direction

Start by reading this proposal alongside parent issue #8275 and the related issues #9646, #8403, and #9411; no implementation files or tests are identified. Work cannot begin until maintainers decide and record the required ownership, contract, adapter, security, runtime, and related-issue dispositions; implementation issues follow only after acceptance.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, authentication, backend-api-design, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.