grafana / grafana/pyroscope

Demangle native symbols server-side instead of in each SDK/agent

Open
#5,628 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

## Summary

Demangling native symbols is currently the responsibility of each SDK/agent. This issue proposes an **opt-in, read-path** demangling capability in the server, scoped deliberately narrowly so as not to relitigate a decision this repo already made.

## Prior art and prior decisions (please read first)

This ground has been covered, and the conclusion went against server-side name rewriting. Recording it up front:

- #2343 (merged, 2023) replaced Go type parameters with `[...]` server-side. #3010 extended that.
- #2976 (merged, 2024) reverted it, stating plainly: *"The presentation aspects are to be managed on the frontend. Ideally, there should be an option for user to disable all the decorations."* The concrete reason was that rewriting names server-side *"makes it impossible to find the selected node in the source profile in backend."*
- #3940 (merged, 2025) removed the ellipsis stub entirely.
- #3941 (open) states the principle outright: *"We should delegate representation handling to the frontend and always preserve symbols as they are."*
- grafana/profiles-drilldown#420 (closed) is the frontend-side version of the same debate.
- #2479 (merged, 2023) added `optionaly demangle symbols` to the eBPF collector — i.e. the opt-in-flag shape already has precedent here.

**I am not proposing we reverse #2976.** If demangling is presentation, it belongs on the frontend, and that is filed separately as grafana/profiles-drilldown#1122. That issue should probably be considered the primary one.

## What still seems unresolved

Two things survive the #2976 position.

**1. Consumers with no frontend.** `profilecli`, pprof/DOT export, and direct API clients have no presentation layer in which to demangle. Under "backend preserves symbols verbatim, frontend decorates," these consumers get raw Itanium symbols and no recourse whatsoever. A frontend-only fix leaves them out. Note that the DOT renderer already carries a comment anticipating demangling (`pkg/frontend/dot/graph/graph.go:42`), inherited from pprof, which does offer `-symbolize=demangle=`.

**2. Server-side symbolization is where the mangled name is minted.** `pkg/symbolizer` resolves addresses via lidia/debuginfod; lidia returns the raw symbol-table string (`lidia/lidia.go:250`) and the symbolizer writes it into the pprof string table verbatim (`pkg/symbolizer/symbolizer.go:315-319`):

```go
nameIdx, ok := stringMap[line.FunctionName]
if !ok {
nameIdx = int64(len(profile.StringTable))
profile.StringTable = append(profile.StringTable, line.FunctionName)
stringMap[line.FunctionName] = nameIdx
}
```

Emitting the raw symbol here is arguably *correct* under #2976. Worth confirming that's intentional rather than incidental, since it means no agent-side flag can ever affect natively symbolized profiles.

## Why the per-SDK approach alone is not sufficient

- #3982 — a user set `demangle = "full"` on `pyroscope.ebpf` and Rust frames were still mangled.
- grafana/pyroscope-rs#580 — the Rust SDK silently regressed C++ demangling between 2.0.4 and 2.1.1. `symbolic-demangle` was declared with `default-features = false, features = ["rust"]`, so `try_demangle_cpp` compiled to `None` and C++ identifiers passed through untouched. The call site looked correct; only a user diffing two production versions caught it. Fixed opt-in in grafana/pyroscope-rs#581.
- Agent-side demangling also spends CPU and binary size inside the process being profiled, which is why grafana/pyroscope-rs#581 had to default to off.

The pattern is N SDKs and collectors times M mangling schemes (Itanium, MSVC, Rust legacy, Rust v0, Swift), each an independent place this can break invisibly.

## Concrete proposal

An **opt-in, off-by-default** demangling option on the read path, in the spirit of #2479 and pprof's `-symbolize=demangle=`:

- Default behaviour unchanged: raw symbols, preserving #2976 and the node-matching property it protects.
- When explicitly requested, demangle Itanium, MSVC, Rust legacy + v0, and Swift.
- Verbosity levels rather than a boolean (`none|simplified|templates|full`), matching Alloy's existing vocabulary.
- Stored data stays raw, so nothing is baked in and the demangler can be fixed without reingesting.

## Open questions

- Is an opt-in read-path flag acceptable under the #2976 position, given the default stays raw? My read is yes, because the objection there was to unconditional lossy rewriting breaking node lookup.
- Should this instead live only in `profilecli` as a client-side formatting step, leaving the server untouched entirely? That would be the most conservative option and may well be the right one.
- Does #3941 want to be generalized into "the backend never decorates, and every consumer owns presentation," with this issue and grafana/profiles-drilldown#1122 as its two halves?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the prior decisions in #2976 and #3941, then inspect pkg/symbolizer/symbolizer.go, lidia/lidia.go, and pkg/frontend/dot/graph/graph.go. Trace how raw symbols enter profiles and compare the proposed server read-path option with the profilecli-only alternative. Done requires an agreed scope that preserves default raw symbols and identifies the affected consumer paths and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.