oxidecomputer / oxidecomputer/lmar
`lmar -p` thread panics with node "instance not set"
Open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 5
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Observed on 0XV2:913-0000023:002:2AM0YM3H on oxygen manufacturing station.
See /staff/adam/mfg-145/0XV2:913-0000023:002:22NH8XX7 for details.
lmar -p ran successfully on prior boots.
On the current affected boot, every run of lmar -p produces the same error.
Related to:
-
https://github.com/oxidecomputer/mfg-troubleshooting/issues/167
-
https://github.com/oxidecomputer/lmar/blob/main/src/main.rs#L2481
unknown # RUST_BACKTRACE=full ./lmar -p
[0/1/3] pcieb0 -- 0x1022 0x153e -- Gen4x4 SUPPORTED
[2/0/0] nvme0 -- 0x1344 0x51c3 -- Gen4x4 SUPPORTED
[20/1/3] pcieb2 -- 0x1022 0x153e -- Gen4x4 SUPPORTED
[21/0/0] nvme1 -- 0x1344 0x51c3 -- Gen4x4 SUPPORTED
[40/1/1] pcieb3 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[43/0/0] nvme2 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[40/1/2] pcieb4 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[44/0/0] nvme3 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[40/1/3] pcieb5 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[45/0/0] nvme4 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[60/1/1] pcieb8 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[61/0/0] nvme-1 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[60/1/2] pcieb9 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[62/0/0] nvme6 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[60/1/3] pcieb10 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[63/0/0] nvme7 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[a0/1/1] pcieb11 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[a2/0/0] nvme8 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[a0/1/2] pcieb12 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[a3/0/0] nvme9 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[a0/1/3] pcieb13 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[a4/0/0] nvme10 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[a0/1/4] pcieb14 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
[a5/0/0] nvme11 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
[c0/1/1] pcieb16 -- 0x1022 0x153e -- Gen1x8
[c1/0/0] t6mfg1 -- 0x1425 0xb000 -- Gen1x8
[c1/0/1] ??-1 -- 0x1425 0xb001 -- Gen1x8
[c1/0/2] ??-1 -- 0x1425 0xb002 -- Gen1x8
[c1/0/3] ??-1 -- 0x1425 0xb003 -- Gen1x8
[c1/0/4] ??-1 -- 0x1425 0xb004 -- Gen1x8
[c1/0/5] ??-1 -- 0x1425 0xb005 -- Gen1x8
[c1/0/6] ??-1 -- 0x1425 0xb006 -- Gen1x8
[c1/0/7] ??-1 -- 0x1425 0xb007 -- Gen1x8
[e0/1/2] pcieb17 -- 0x1022 0x153e -- Gen3x4
[e3/0/0] ixgbe0 -- 0x8086 0x1563 -- Gen3x4
[e3/0/1] ixgbe1 -- 0x8086 0x1563 -- Gen3x4
---
Margining [2/0/0] nvme0 -- 0x1344 0x51c3 -- Gen4x4 SUPPORTED
Margining [43/0/0] nvme2 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [40/1/1] pcieb3 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [21/0/0] nvme1 -- 0x1344 0x51c3 -- Gen4x4 SUPPORTED
Margining [40/1/2] pcieb4 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [0/1/3] pcieb0 -- 0x1022 0x153e -- Gen4x4 SUPPORTED
Margining [45/0/0] nvme4 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [20/1/3] pcieb2 -- 0x1022 0x153e -- Gen4x4 SUPPORTED
Margining [40/1/3] pcieb5 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [61/0/0] nvme-1 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [60/1/1] pcieb8 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [44/0/0] nvme3 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [62/0/0] nvme6 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [60/1/2] pcieb9 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [63/0/0] nvme7 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [a2/0/0] nvme8 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [60/1/3] pcieb10 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [a0/1/1] pcieb11 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [a3/0/0] nvme9 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [a0/1/2] pcieb12 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [a4/0/0] nvme10 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [a0/1/3] pcieb13 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
Margining [a5/0/0] nvme11 -- 0x1b96 0x2751 -- Gen5x4 SUPPORTED
Margining [a0/1/4] pcieb14 -- 0x1022 0x153e -- Gen5x4 SUPPORTED
thread '<unnamed>' (12) panicked at src/main.rs:2481:51:
instance not set
stack backtrace:
0: 0x80659e - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h7fb787cf02ecf067
1: 0x8376cf - core::fmt::write::h0e6ffda62e2ab989
2: 0x7bf010 - std::io::Write::write_fmt::hf87b20e1b5abcc1d
3: 0x7d0a12 - std::sys::backtrace::BacktraceLock::print::h2118728da495bf95
4: 0x7d73dc - std::panicking::default_hook::{{closure}}::ha6ef311b4a961ac9
5: 0x7d7226 - std::panicking::default_hook::h09afcfa093540a53
6: 0x7d7b2b - std::panicking::panic_with_hook::h6f6630bb15543974
7: 0x7d78d8 - std::panicking::panic_handler::{{closure}}::h0fa8884cef8a0651
8: 0x7d0b49 - std::sys::backtrace::__rust_end_short_backtrace::h25213986b109b8dd
9: 0x7b1f0c - __rustc[eb8946e36839644a]::rust_begin_unwind
10: 0x847c7f - core::panicking::panic_fmt::hda720bd8f33a9fd1
11: 0x84654a - core::option::expect_failed::h1140a1161c0e28b2
12: 0x569847 - lmar::margin_one::hbcb088101b6b4fe6
13: 0x5810f9 - std::sys::backtrace::__rust_begin_short_backtrace::h2cf525cbd8a59a52
14: 0x5ad0c5 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hce4fe0e9a70597e5
15: 0x7ccc3f - std::sys::thread::unix::Thread::new::thread_start::ha097fb815024401f
16: 0xfffff1ffeeb27027 - _thrp_setup
17: 0xfffff1ffeeb27370 - <unknown>
^C
unknown #
unknown # /usr/lib/pci/pcieadm show-devs
BDF TYPE INSTANCE DEVICE
0/0/0 PCI amdzen_stub16 Turin Root Complex
0/0/2 PCI -- Turin IOMMU
0/0/3 PCI -- Turin RCEC
0/1/0 PCI -- Turin PCIe Dummy Host Bridge
0/1/3 PCIe Gen 4x4 pcieb0 Turin GPP Bridge
2/0/0 PCIe Gen 4x4 nvme0 7450 PRO NVMe SSD
0/2/0 PCI -- Turin PCIe Dummy Host Bridge
0/3/0 PCI -- Turin PCIe Dummy Host Bridge
0/4/0 PCI -- Turin PCIe Dummy Host Bridge
0/5/0 PCI -- Turin PCIe Dummy Host Bridge
0/7/0 PCI -- Turin PCIe Dummy Host Bridge
0/7/1 PCIe Gen 5x16 pcieb1 Turin Internal PCIe GPP Bridge to Bus [D:C]
1/0/0 PCIe Gen 5x16 -- Turin PCIe Dummy Function
0/14/0 PCI -- FCH SMBus Controller
0/14/3 PCI -- FCH LPC Bridge
0/18/0 PCI amdzen_stub17 Turin Data Fabric; Function 0
0/18/1 PCI amdzen_stub18 Turin Data Fabric; Function 1
0/18/2 PCI amdzen_stub19 Turin Data Fabric; Function 2
0/18/3 PCI amdzen_stub20 Turin Data Fabric; Function 3
0/18/4 PCI amdzen_stub21 Turin Data Fabric; Function 4
0/18/5 PCI amdzen_stub22 Turin Data Fabric; Function 5
0/18/6 PCI amdzen_stub23 Turin Data Fabric; Function 6
0/18/7 PCI amdzen_stub24 Turin Data Fabric; Function 7
20/0/0 PCI amdzen_stub25 Turin Root Complex
20/0/3 PCI -- Turin RCEC
20/1/0 PCI -- Turin PCIe Dummy Host Bridge
20/1/3 PCIe Gen 4x4 pcieb2 Turin GPP Bridge
21/0/0 PCIe Gen 4x4 nvme1 7450 PRO NVMe SSD
20/2/0 PCI -- Turin PCIe Dummy Host Bridge
40/0/0 PCI amdzen_stub30 Turin Root Complex
BDF TYPE INSTANCE DEVICE
40/0/2 PCI -- Turin IOMMU
40/0/3 PCI -- Turin RCEC
40/1/0 PCI -- Turin PCIe Dummy Host Bridge
40/1/1 PCIe Gen 5x4 pcieb3 Turin GPP Bridge
43/0/0 PCIe Gen 5x4 nvme2 Ultrastar DC SN861 NVMe SSD
40/1/2 PCIe Gen 5x4 pcieb4 Turin GPP Bridge
44/0/0 PCIe Gen 5x4 nvme3 Ultrastar DC SN861 NVMe SSD
40/1/3 PCIe Gen 5x4 pcieb5 Turin GPP Bridge
45/0/0 PCIe Gen 5x4 nvme4 Ultrastar DC SN861 NVMe SSD
40/2/0 PCI -- Turin PCIe Dummy Host Bridge
40/3/0 PCI -- Turin PCIe Dummy Host Bridge
40/4/0 PCI -- Turin PCIe Dummy Host Bridge
40/5/0 PCI -- Turin PCIe Dummy Host Bridge
40/7/0 PCI -- Turin PCIe Dummy Host Bridge
40/7/1 PCIe Gen 5x16 pcieb6 Turin Internal PCIe GPP Bridge to Bus [D:C]
41/0/0 PCIe Gen 5x16 -- Turin PCIe Dummy Function
41/0/2 PCIe Gen 5x16 -- Unknown device: 0x14c0
41/0/3 PCIe Gen 5x16 -- Secondary vNTB
41/0/4 PCIe Gen 5x16 -- Turin USB 3.1 xHCI
41/0/5 PCIe Gen 5x16 -- Turin CCP/ASP
41/0/6 PCIe Gen 5x16 -- Unknown device: 0x14cb
41/0/7 PCIe Gen 5x16 -- Unknown device: 0x14cc
40/7/2 PCIe Gen 5x16 pcieb7 Turin Internal PCIe GPP Bridge to Bus [D:C]
42/0/0 PCIe Gen 5x16 -- FCH SATA Controller [AHCI mode]
42/0/1 PCIe Gen 5x16 -- FCH SATA Controller [AHCI mode]
60/0/0 PCI amdzen_stub27 Turin Root Complex
60/0/3 PCI -- Turin RCEC
60/1/0 PCI -- Turin PCIe Dummy Host Bridge
60/1/1 PCIe Gen 5x4 pcieb8 Turin GPP Bridge
61/0/0 PCIe Gen 5x4 -- Ultrastar DC SN861 NVMe SSD
BDF TYPE INSTANCE DEVICE
60/1/2 PCIe Gen 5x4 pcieb9 Turin GPP Bridge
62/0/0 PCIe Gen 5x4 nvme6 Ultrastar DC SN861 NVMe SSD
60/1/3 PCIe Gen 5x4 pcieb10 Turin GPP Bridge
63/0/0 PCIe Gen 5x4 nvme7 Ultrastar DC SN861 NVMe SSD
60/2/0 PCI -- Turin PCIe Dummy Host Bridge
80/0/0 PCI amdzen_stub28 Turin Root Complex
80/0/3 PCI -- Turin RCEC
80/1/0 PCI -- Turin PCIe Dummy Host Bridge
80/2/0 PCI -- Turin PCIe Dummy Host Bridge
a0/0/0 PCI amdzen_stub26 Turin Root Complex
a0/0/2 PCI -- Turin IOMMU
a0/0/3 PCI -- Turin RCEC
a0/1/0 PCI -- Turin PCIe Dummy Host Bridge
a0/1/1 PCIe Gen 5x4 pcieb11 Turin GPP Bridge
a2/0/0 PCIe Gen 5x4 nvme8 Ultrastar DC SN861 NVMe SSD
a0/1/2 PCIe Gen 5x4 pcieb12 Turin GPP Bridge
a3/0/0 PCIe Gen 5x4 nvme9 Ultrastar DC SN861 NVMe SSD
a0/1/3 PCIe Gen 5x4 pcieb13 Turin GPP Bridge
a4/0/0 PCIe Gen 5x4 nvme10 Ultrastar DC SN861 NVMe SSD
a0/1/4 PCIe Gen 5x4 pcieb14 Turin GPP Bridge
a5/0/0 PCIe Gen 5x4 nvme11 Ultrastar DC SN861 NVMe SSD
a0/2/0 PCI -- Turin PCIe Dummy Host Bridge
a0/3/0 PCI -- Turin PCIe Dummy Host Bridge
a0/4/0 PCI -- Turin PCIe Dummy Host Bridge
a0/5/0 PCI -- Turin PCIe Dummy Host Bridge
a0/7/0 PCI -- Turin PCIe Dummy Host Bridge
a0/7/1 PCIe Gen 5x16 pcieb15 Turin Internal PCIe GPP Bridge to Bus [D:C]
a1/0/0 PCIe Gen 5x16 -- Turin PCIe Dummy Function
a1/0/2 PCIe Gen 5x16 -- Unknown device: 0x14c0
a1/0/3 PCIe Gen 5x16 -- Secondary vNTB
BDF TYPE INSTANCE DEVICE
a1/0/4 PCIe Gen 5x16 -- Turin USB 3.1 xHCI
a1/0/5 PCIe Gen 5x16 -- Turin CCP/ASP
a1/0/6 PCIe Gen 5x16 -- Unknown device: 0x14cb
a1/0/7 PCIe Gen 5x16 -- Unknown device: 0x14cc
c0/0/0 PCI amdzen_stub31 Turin Root Complex
c0/0/3 PCI -- Turin RCEC
c0/1/0 PCI -- Turin PCIe Dummy Host Bridge
c0/1/1 PCIe Gen 1x8 pcieb16 Turin GPP Bridge
c1/0/0 PCIe Gen 1x8 t6mfg1 Unknown device: 0xb000
c1/0/1 PCIe Gen 1x8 -- Unknown device: 0xb001
c1/0/2 PCIe Gen 1x8 -- Unknown device: 0xb002
c1/0/3 PCIe Gen 1x8 -- Unknown device: 0xb003
c1/0/4 PCIe Gen 1x8 -- Unknown device: 0xb004
c1/0/5 PCIe Gen 1x8 -- Unknown device: 0xb005
c1/0/6 PCIe Gen 1x8 -- Unknown device: 0xb006
c1/0/7 PCIe Gen 1x8 -- Unknown device: 0xb007
c0/2/0 PCI -- Turin PCIe Dummy Host Bridge
e0/0/0 PCI amdzen_stub29 Turin Root Complex
e0/0/2 PCI -- Turin IOMMU
e0/0/3 PCI -- Turin RCEC
e0/1/0 PCI -- Turin PCIe Dummy Host Bridge
e0/1/2 PCIe Gen 3x4 pcieb17 Turin GPP Bridge
e3/0/0 PCIe Gen 3x4 ixgbe0 Ethernet Controller X550
e3/0/1 PCIe Gen 3x4 ixgbe1 Ethernet Controller X550
e0/2/0 PCI -- Turin PCIe Dummy Host Bridge
e0/3/0 PCI -- Turin PCIe Dummy Host Bridge
e0/4/0 PCI -- Turin PCIe Dummy Host Bridge
e0/5/0 PCI -- Turin PCIe Dummy Host Bridge
e0/7/0 PCI -- Turin PCIe Dummy Host Bridge
e0/7/1 PCIe Gen 5x16 pcieb18 Turin Internal PCIe GPP Bridge to Bus [D:C]
BDF TYPE INSTANCE DEVICE
e1/0/0 PCIe Gen 5x16 -- Turin PCIe Dummy Function
e0/7/2 PCIe Gen 5x16 pcieb19 Turin Internal PCIe GPP Bridge to Bus [D:C]
e2/0/0 PCIe Gen 5x16 -- FCH SATA Controller [AHCI mode]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/main.rs:2481 and trace the margin_one path invoked by lmar -p. Reproduce with the reported PCIe device output and compare it with /usr/lib/pci/pcieadm show-devs, focusing on the device whose instance is unset. Done means lmar -p completes without panicking when this hardware state is present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100