NVIDIA / NVIDIA/nvrc

How to build NVRC locally?

Open
#202 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
45
Forks
22
Avg merge
8h 44m
Merged PRs (30d)
6

Description

Background

In order to debug NVRC panic issue for 4 or 8 GPU passthrough scenario,I tried to increase timeout value(timeout waiting for: Local RPC services initialized) and build NVRC locally, the build process is fine, but after I replace NVRC binary in kata 3.32.0 guest image with my local built one, vm booting process always failed due to init process issue, I tried different building machines and different building OS distros, no lucky.
And I can boot vm successfully by replacing NVRC binary with the github released artifacts.

Building command

# for v0.1.5 version
./scripts/build-release.sh x86_64-unknown-linux-musl

# for v0.1.4 version
cargo build --release --target x86_64-unknown-linux-musl

Boot failed logs for Ubuntu 25.04 building machine

# got werid binary, should expect statically linked
root@0714:~/nvrc# file target/x86_64-unknown-linux-musl/release/NVRC
target/x86_64-unknown-linux-musl/release/NVRC: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, stripped


# failed booting logs
1491:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.314621] devtmpfs: mounted"
1492:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.315530] Freeing unused decrypted memory: 2036K"
1493:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.316237] Freeing unused kernel image (initmem) memory: 1292K"
1494:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.316330] Write protecting the kernel read-only data: 18432k"
1495:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.317533] Freeing unused kernel image (text/rodata gap) memory: 1292K"
1496:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.318209] Freeing unused kernel image (rodata/data gap) memory: 884K"
1497:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.318348] Run /sbin/init as init process"
1498:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.320732] Run /etc/init as init process"
1499:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.321027] Run /bin/init as init process"
1500:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "[   10.321140] Run /bin/sh as init process"
1501:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "sh: cannot set terminal process group (-1): Inappropriate ioctl for device"
1502:Jul 18 18:29:09 g1061 kata[3671077]: vm console: "sh: no job control in this shell"
1829:Jul 18 18:29:25 g1061 kata[3671077]: mgmt-svr(mux): recv req, method: GET, uri: /metrics
1874:Jul 18 18:29:28 g1061 kata[3671077]: Failed to serve connection: hyper::Error(IncompleteMessage)

Boot failed logs for Centos 9 building machine

# got a normal binary
[root@g1061 nvrc]# file target/x86_64-unknown-linux-musl/release/NVRC
target/x86_64-unknown-linux-musl/release/NVRC: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

# failed booting logs
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.875560] Freeing unused decrypted memory: 2036K"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.876228] Freeing unused kernel image (initmem) memory: 1292K"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.876342] Write protecting the kernel read-only data: 18432k"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.877525] Freeing unused kernel image (text/rodata gap) memory: 1292K"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.878186] Freeing unused kernel image (rodata/data gap) memory: 884K"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.878307] Run /sbin/init as init process"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.880711] init[1]: segfault at 0 ip 0000000000409a8d sp 00007ffc683f4558 error 4 in NVRC-x86_64-unknown-linux-musl[9a8d,401000+67000] likely on CPU 0 (core 0, socket 0)"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.880946] Code: 4a 08 4c 89 4c cc 88 eb e3 48 8d 94 24 b0 01 00 00 48 c7 00 00 00 00 00 48 83 c0 08 48 39 d0 75 f0 48 89 f2 eb 04 48 83 c2 10 <48> 8b 02 48 85 c0 74 14 48 83 f8 24 77 ee 48 8b 4a 08 48 89 8c c4"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881215] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881329] CPU: 0 UID: 0 PID: 1 Comm: init Not tainted 6.18.35-nvidia-gpu #14 NONE "
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881444] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881555] Call Trace:"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881596]  <TASK>"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881634]  vpanic+0x2ec/0x300"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881704]  panic+0x52/0x60"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881776]  do_exit+0x7c4/0x960"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881845]  do_group_exit+0x27/0x80"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881903]  get_signal+0x744/0x750"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.881964]  arch_do_signal_or_restart+0x24/0x1b0"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.882044]  ? force_sig_fault+0x43/0x60"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.882102]  ? exc_page_fault+0x27f/0x6b0"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.882168]  irqentry_exit_to_user_mode+0xc8/0x230"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.882246]  asm_exc_page_fault+0x22/0x30"
Jul 18 18:33:52 g1061 kata[3709465]: vm console: "[   10.882303] RIP: 0033:0x409a8d"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/build-release.sh and the cargo build command for x86_64-unknown-linux-musl, then compare the resulting NVRC file type with the Ubuntu and CentOS examples. Reproduce the guest boot using the locally built binary and inspect the init segfault or dynamic-linking failure; done means the local build produces an artifact that boots successfully in the reported Kata guest scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.