mlcommons / mlcommons/inference
Inference [4.1.5] Inference nvidia-smi command exits but failed to execute
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
Hello,
I ran the following command: nohup make generate_engines RUN_ARGS="--benchmarks=stable-diffusion-xl --scenarios=offline" 2>&1 | tee -a /Inference_Store/mlperf_4.1.5_inference_data/mlperf-logs/generate_engines_offline
And ran into the following errors.
[2025-02-06 15:29:21,626 systems.py:174 WARNING] nvidia-smi command exists but failed to execute - Ignoring NUMA detection.
[2025-02-06 15:29:21,870 main.py:227 INFO] Detected system did not match any known systems. Exiting. SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='Cortex-A78AE', architecture=<CPUArchitecture.aarch64: AliasedName(name='aarch64', aliases=(), patterns=())>, core_count=4, threads_per_core=1): 1}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=64.348844, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=64348844000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='CTI Forge + Orin AGX', accelerator_type=<AcceleratorType.Integrated: AliasedName(name='Integrated', aliases=(), patterns=())>, vram=None, max_power_limit=None, pci_id=None, compute_sm=87): 1})), numa_conf=None, system_id=None)
(mlperf) nvidia@mlperf-inference-nvidia-aarch64-1018:/closed/NVIDIA$ nvidia-smi
Thu Feb 6 15:34:15 2025
Any advice on how to resolve this issue? How can I ensure my system is detected and how can I check what my known systems are.
NVIDIA-SMI version: 540.4.0
Cuda version: 12.6
Driver version 540.4.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with systems.py:174, where the nvidia-smi warning is emitted, and main.py:227, where unmatched system configurations exit. Reproduce the make generate_engines command and inspect how known systems are represented and selected for the reported CPU, memory, and accelerator configuration. Done means the reported system is either detected correctly or the unsupported case is clearly handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100