oxidecomputer / oxidecomputer/omicron

Why do these Nexus core files not show some function names in their stack traces?

Open
#5,359 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

During today's mupdate, we saw a couple of Nexus cores on gc08 which have been rsynced over to /staff/dock/rack2/mupdate-20240329/cores/sled-08. With both those cores, whether you do pstack or mdb with $c, you get something like:

libc.so.1`_lwp_kill+0xa()
libc.so.1`raise+0x22(6)
libc.so.1`abort+0x58()
0x5a207b9()
0x5a207a9()
rust_panic+0xd()
_ZN3std9panicking20rust_panic_with_hook17h7d19ef586749da2fE+0x2ab()
_ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h95e3e1ca24551a68E+0xa4()
0x5a05ef9()
0x5a08802()
0x5a4f6c5()
0x5a4fd55()
_ZN95_$LT$nexus_db_queries..db..sec_store..CockroachDbSecStore$u20$as$u20$steno..store..SecStore$GT$12record_event28_$u7b$$u7b$closure$u7d$$u7d$17hc33ebb1cf636f348E+0xcec()
...

Why do the functions in the middle (0x5a05ef9 etc) not have their names in the stack traces? Maybe they're in std -- but if so, why does std::panicking::begin_panic_handler get shown?

I'm concerned that this issue may be more widespread somehow.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the two cores under /staff/dock/rack2/mupdate-20240329/cores/sled-08 and reproduce the traces using pstack and mdb with $c. Compare the unnamed 0x5a05ef9-style frames with the named Rust frames. Done means explaining why those function names are missing and whether the issue is limited to these cores or more widespread.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.