NVIDIA / NVIDIA/NemoClaw

Performance test Nemotron Ultra 3 with Deep Code

Open
#5,830 2 comments 0 reactions 1 assignee Claimed by @ahunnargikar-nvidia View on GitHub
area: inference area: performance enhancement integration: dcode needs: design provider: nvidia
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Goal

Measure and document the performance of Deep Code running on Nemotron Ultra 3 for representative coding-agent workflows.

## Scope

- Measure end-to-end coding-agent performance, including:
- time to first useful response
- total task completion time
- tool-call round-trip latency
- model response latency and throughput where available
- timeout, retry, and failure rates
- stability across multi-step coding tasks
- Capture the test environment, model configuration, provider path, and relevant runtime settings.
- Identify bottlenecks in inference, routing, orchestration, tool execution, context handling, or retries.
- Document recommended tuning or follow-up work.

## Acceptance Criteria

- A repeatable performance test plan exists for Deep Code on Nemotron Ultra 3.
- At least one representative multi-step coding workflow is measured end to end.
- Results include latency, stability, and failure-rate observations.
- The test records enough configuration and environment detail to reproduce the run.
- The final summary states whether Deep Code on Nemotron Ultra 3 meets the expected performance bar.
- Any discovered performance gaps are filed or linked as follow-up issues.

## Non-Goals

- Do not benchmark every supported model or provider.
- Do not rely only on synthetic token throughput.
- Do not require CI automation in the first version unless promoted into follow-up scope.

## Related Work

- #3123
- #5604
- #3776

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.