grafana / grafana/pyroscope

Support excluding stack frames by function name in flame graph

Open
#4,915 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Is your feature request related to a problem? Please describe.

Off-CPU flame graphs for Go programs are dominated by expected runtime behavior: `runtime.futex` (synchronization) and `runtime.usleep` (sysmon housekeeping). These frames bury the actual application-level blocking signal, making it hard to identify real bottlenecks like I/O waits or lock contention.

Pyroscope's search highlights matching frames but does not remove non-matching frames or adjust totals. Focus Block and Sandwich View help investigate specific subtrees or functions but don't address broad noise removal across the entire flame graph.

#### Describe the solution you'd like

Query-time filters that exclude stack frames by function name, with two modes:

1. **Stack filter**: if any frame in a stack matches, drop the entire sample. For removing stacks dominated by idle functions (e.g., `runtime.futex`).
2. **Frame filter**: hide individual matching frames while preserving the rest of the stack and its sample value. For hiding library/runtime internals without losing call context.

Filtering should happen at query time only (stored data is unmodified). The response should include both filtered and unfiltered totals so the UI can show accurate percentages relative to either.

Built-in presets for common patterns would reduce friction:
- "Go Runtime Off-CPU": excludes `runtime.futex`, `runtime.usleep`
- Similar presets for other runtimes (Rust tokio, Python, etc.)

#### Describe alternatives you've considered

- **Alloy-level filtering**: drop stack frames matching known patterns in the agent before sending to Pyroscope. This permanently destroys data and prevents retroactive analysis with different filters.
- **Post-hoc scripting**: export profiles and filter externally. Loses the interactive flame graph experience and doesn't scale.
- **Using Parca for off-CPU analysis**: pragmatic but adds operational overhead of running two profiling backends.

#### Additional context

Parca implements this server-side with two filter types (stack and frame), string conditions (equal, not_equal, contains, not_contains, starts_with, not_starts_with), and UI-defined presets that expand into filter API calls:

- Presets: [`filterPresets.ts`](https://github.com/parca-dev/parca/blob/main/ui/packages/shared/profile/src/ProfileView/components/ProfileFilters/filterPresets.ts)
- Query API: [`query.proto`](https://github.com/parca-dev/parca/blob/main/proto/parca/query/v1alpha1/query.proto) (`StackFilter`, `FrameFilter`, `StringCondition`)
- Backend: [`columnquery.go`](https://github.com/parca-dev/parca/blob/main/pkg/query/columnquery.go) (`FilterProfileData`)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the query API and backend examples linked in Parca's query.proto and columnquery.go, then compare them with Pyroscope's existing query and flame-graph paths. The UI preset reference is filterPresets.ts. Done means query-time stack and frame filtering, filtered and unfiltered totals, and the requested preset behavior are covered across the relevant API, backend, and UI surfaces.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.