mlcommons / mlcommons/inference

Inference [4.1.5] Inference nvidia-smi command exits but failed to execute

Open
#2,098 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.6k
Forks
650
Avg merge
1d 22h
Merged PRs (30d)
6

Description

Hello,

I ran the following command: nohup make generate_engines RUN_ARGS="--benchmarks=stable-diffusion-xl --scenarios=offline" 2>&1 | tee -a /Inference_Store/mlperf_4.1.5_inference_data/mlperf-logs/generate_engines_offline

And ran into the following errors.
[2025-02-06 15:29:21,626 systems.py:174 WARNING] nvidia-smi command exists but failed to execute - Ignoring NUMA detection.
[2025-02-06 15:29:21,870 main.py:227 INFO] Detected system did not match any known systems. Exiting. SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='Cortex-A78AE', architecture=<CPUArchitecture.aarch64: AliasedName(name='aarch64', aliases=(), patterns=())>, core_count=4, threads_per_core=1): 1}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=64.348844, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=64348844000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='CTI Forge + Orin AGX', accelerator_type=<AcceleratorType.Integrated: AliasedName(name='Integrated', aliases=(), patterns=())>, vram=None, max_power_limit=None, pci_id=None, compute_sm=87): 1})), numa_conf=None, system_id=None)
(mlperf) nvidia@mlperf-inference-nvidia-aarch64-1018:/closed/NVIDIA$ nvidia-smi
Thu Feb  6 15:34:15 2025

Any advice on how to resolve this issue? How can I ensure my system is detected and how can I check what my known systems are.

NVIDIA-SMI version: 540.4.0
Cuda version: 12.6
Driver version 540.4.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with systems.py:174, where the nvidia-smi warning is emitted, and main.py:227, where unmatched system configurations exit. Reproduce the make generate_engines command and inspect how known systems are represented and selected for the reported CPU, memory, and accelerator configuration. Done means the reported system is either detected correctly or the unsupported case is clearly handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.