Performance test Nemotron Ultra 3 with Deep Code
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Goal
Measure and document the performance of Deep Code running on Nemotron Ultra 3 for representative coding-agent workflows.
## Scope
- Measure end-to-end coding-agent performance, including:
- time to first useful response
- total task completion time
- tool-call round-trip latency
- model response latency and throughput where available
- timeout, retry, and failure rates
- stability across multi-step coding tasks
- Capture the test environment, model configuration, provider path, and relevant runtime settings.
- Identify bottlenecks in inference, routing, orchestration, tool execution, context handling, or retries.
- Document recommended tuning or follow-up work.
## Acceptance Criteria
- A repeatable performance test plan exists for Deep Code on Nemotron Ultra 3.
- At least one representative multi-step coding workflow is measured end to end.
- Results include latency, stability, and failure-rate observations.
- The test records enough configuration and environment detail to reproduce the run.
- The final summary states whether Deep Code on Nemotron Ultra 3 meets the expected performance bar.
- Any discovered performance gaps are filed or linked as follow-up issues.
## Non-Goals
- Do not benchmark every supported model or provider.
- Do not rely only on synthetic token throughput.
- Do not require CI automation in the first version unless promoted into follow-up scope.
## Related Work
- #3123
- #5604
- #3776
Contributor guide
Assessment
This issue has not been assessed yet.