oxidecomputer / oxidecomputer/usdt

stapsdt, monormoprhisation, and linker --gc-sections.

Open
#498 11 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
145
Forks
15
Avg merge
14m
Merged PRs (30d)
6

Description

I've spent a few days trying to debug why I'm getting weird behaviour with usdt on linux.

What I'm observing is many stapsdt notes that end up linked into my ELF file have the probe address unresolved. The conditions for this bug to appear are quite hard to minimise for:

  1. It requires a probe to be part of a monomorphised function
  2. The monomorphised function needs to be instantiated across multiple codegen units
  3. Using the lld or mold linker

By eliminating any of these the behaviour would not manifest. After some time chatting with peers. I was able to find that the bug disappears with RUSTFLAGS=-Clink-args=-Wl,--no-gc-sections. This leads me to conclude the bug is the following:

  1. A function is monomorphised across several CGU.
  2. This monomorphised function contains the stapsdt note.
  3. The linker deduplicates these functions when applying GC
  4. The linker does not remove any dangling notes.

I've found 2 ways to resolve the issue in my testing. But I don't understand whether there's any other consequences of the code they produce. They both relate to the following line

https://github.com/oxidecomputer/usdt/blob/4329591597b3ceaab78d606cf601781c6a99e6de/usdt-impl/src/stapsdt.rs#L156

Retained

The first fix is to mark the note as "retained". This ensures that the functions don't get de-duplicated and thus the notes do not dangle. This is not ideal as this causes binary bloat.

        .pushsection .note.stapsdt, "R", "note"
Comdat

The second fix I don't fully understand but it works in all my testing and I can't find any problems with the resulting ELF. It uses the following change:

        .pushsection .note.stapsdt, "G", "note", ".note.stapsdt.{prov}.{probe}", comdat

cc @aapoalas

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at usdt-impl/src/stapsdt.rs line 156 and reproduce the unresolved probe addresses with monomorphised functions, multiple codegen units, lld or mold, and --gc-sections. Compare the retained and Comdat section variants, then verify that the chosen behavior prevents dangling stapsdt notes without unintended ELF or binary-size consequences.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
build-system, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.