mlcommons / mlcommons/inference

sudo access needed to run cm run script "get sys-utils-cm"

Open
#1,846 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.6k
Forks
650
Avg merge
1d 22h
Merged PRs (30d)
6

Description

Hi,

This is my attached cm-repro file
cm-repro.zip

I'm trying to run the MLPerf Reference Implementation for bert-large at https://docs.mlcommons.org/inference/benchmarks/language/bert/ and I'm running into issues while trying to run

cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev \
   --model=bert-99 \
   --implementation=reference \
   --framework=pytorch \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cpu  \
   --quiet \
   --test_query_count=100

I see in the terminal that it's asking me for my sudo password, but I don't have sudo access on this cluster. Here's the output:

INFO:root:* cm run script "run-mlperf inference _find-performance _full _r4.1-dev"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /u1/wk5ng/finetune-inference-experiments
INFO:root:         ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "detect cpu"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /u1/wk5ng/finetune-inference-experiments
INFO:root:           ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:         ! cd /u1/wk5ng/finetune-inference-experiments
INFO:root:         ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:  * cm run script "get python3"
INFO:root:       ! load /u1/wk5ng/CM/repos/local/cache/4f9e6fe06cd94940/cm-cached-state.json
INFO:root:Path to Python: /u1/wk5ng/CM/repos/local/cache/7de49cf355794755/mlperf/bin/python3
INFO:root:Python version: 3.11.4
INFO:root:  * cm run script "get mlcommons inference src"
INFO:root:       ! load /u1/wk5ng/CM/repos/local/cache/639bf66ba64d46e2/cm-cached-state.json
INFO:root:  * cm run script "get sut description"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /u1/wk5ng/finetune-inference-experiments
INFO:root:           ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /u1/wk5ng/finetune-inference-experiments
INFO:root:             ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /u1/wk5ng/finetune-inference-experiments
INFO:root:           ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /u1/wk5ng/CM/repos/local/cache/4f9e6fe06cd94940/cm-cached-state.json
INFO:root:Path to Python: /u1/wk5ng/CM/repos/local/cache/7de49cf355794755/mlperf/bin/python3
INFO:root:Python version: 3.11.4
INFO:root:    * cm run script "get compiler"
INFO:root:         ! load /u1/wk5ng/CM/repos/local/cache/6e0d265390494dc3/cm-cached-state.json
INFO:root:    * cm run script "get generic-python-lib _package.dmiparser"
INFO:root:         ! load /u1/wk5ng/CM/repos/local/cache/1404f366b8494c6e/cm-cached-state.json
INFO:root:    * cm run script "get cache dir _name.mlperf-inference-sut-descriptions"
INFO:root:         ! load /u1/wk5ng/CM/repos/local/cache/b0193ec77b0842dc/cm-cached-state.json
Generating SUT description file for watgpu408-pytorch
INFO:root:         ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-sut-description/customize.py
INFO:root:  * cm run script "get mlperf inference results dir"
INFO:root:       ! load /u1/wk5ng/CM/repos/local/cache/087330972af64098/cm-cached-state.json
INFO:root:  * cm run script "install pip-package for-cmind-python _package.tabulate"
INFO:root:       ! load /u1/wk5ng/CM/repos/local/cache/5ae9aab0c8ea46dd/cm-cached-state.json
INFO:root:  * cm run script "get mlperf inference utils"
INFO:root:    * cm run script "get mlperf inference src"
INFO:root:         ! load /u1/wk5ng/CM/repos/local/cache/639bf66ba64d46e2/cm-cached-state.json
INFO:root:         ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /u1/wk5ng/CM/repos/local/cache/5584207b30dd4fc1/inference

Running loadgen scenario: Offline and mode: performance
INFO:root:* cm run script "app mlperf inference generic _reference _bert-99 _pytorch _cpu _test _r4.1-dev_default _offline"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /u1/wk5ng/finetune-inference-experiments
INFO:root:         ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "get sys-utils-cm"
INFO:root:       ! load /u1/wk5ng/CM/repos/local/cache/69c01310f61e445c/cm-cached-state.json
INFO:root:  * cm run script "get python"
INFO:root:       ! load /u1/wk5ng/CM/repos/local/cache/4f9e6fe06cd94940/cm-cached-state.json
INFO:root:Path to Python: /u1/wk5ng/CM/repos/local/cache/7de49cf355794755/mlperf/bin/python3
INFO:root:Python version: 3.11.4
INFO:root:  * cm run script "get mlcommons inference src _deeplearningexamples"
INFO:root:       ! load /u1/wk5ng/CM/repos/local/cache/639bf66ba64d46e2/cm-cached-state.json
INFO:root:  * cm run script "get mlperf inference utils"
INFO:root:    * cm run script "get mlperf inference src _deeplearningexamples"
INFO:root:         ! load /u1/wk5ng/CM/repos/local/cache/639bf66ba64d46e2/cm-cached-state.json
INFO:root:         ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
INFO:root:  * cm run script "get dataset squad language-processing"
INFO:root:    * cm run script "get sys-utils-cm"
INFO:root:         ! load /u1/wk5ng/CM/repos/local/cache/69c01310f61e445c/cm-cached-state.json
INFO:root:    * cm run script "get sys-util generic generic-sys-util _wget"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /u1/wk5ng/CM/repos/local/cache/ad9e50b30233426f
INFO:root:             ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /u1/wk5ng/CM/repos/local/cache/ad9e50b30233426f
INFO:root:           ! call /u1/wk5ng/CM/repos/mlcommons@cm4mlops/script/get-generic-sys-util/run.sh from tmp-run.sh
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y wget
[sudo] password for wk5ng: 


Sorry, try again.
[sudo] password for wk5ng: 

Sorry, try again.
[sudo] password for wk5ng: 

sudo: 3 incorrect password attempts

CM error: Portable CM script failed (name = get-generic-sys-util, return code = 256)

I see that it's trying to install wget, but when I run wget -h I see that it's installed on the system. Is there anyway to bypass this step?

Thank you so much,
Indy Ng

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported cm run script flow and get-generic-sys-util/run.sh, where the log invokes apt-get to install wget. Reproduce the run on a system where wget is already available but sudo is unavailable, then verify that the MLPerf BERT command completes without requiring sudo.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.