grafana / grafana/pyroscope

Querying unsymbolized profiles with maxNodes !=0 returns truncated/invalid data

Open
#4,789 0 comments 0 reactions 1 assignee Claimed by @korniltsev-grafanista View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

```
Period: 0
Samples:
cpu/nanoseconds
1000000000: 1 2
1000000000: 3 4
1000000000: 5 6
Locations
1: 0x0 M=1 foo :0:0 s=0()
2: 0x0 M=1 bar :0:0 s=0()
3: 0xcafe03000 M=2
4: 0xcafe04000 M=2
5: 0xcafe05000 M=2
6: 0xcafe06000 M=2
Mappings
1: 0x0/0x0/0x0
2: 0x0/0x0/0x0 library.so
```

gprofile go code

```golang
gp := &gprofile.Profile{}
gp.Mapping = []*gprofile.Mapping{
{
ID: 1,
},
{
ID: 2,
File: "library.so",
},
}
gp.Function = []*gprofile.Function{
{
ID: 1,
Name: "foo",
},
{
ID: 2,
Name: "bar",
},
}
gp.Location = []*gprofile.Location{
{
ID: 1,
Mapping: gp.Mapping[0],
Line: []gprofile.Line{
{
Function: gp.Function[0],
},
},
},
{
ID: 2,
Mapping: gp.Mapping[0],
Line: []gprofile.Line{
{
Function: gp.Function[1],
},
},
},
{ID: 3, Mapping: gp.Mapping[1], Address: 0xcafe0_3000},
{ID: 4, Mapping: gp.Mapping[1], Address: 0xcafe0_4000},
{ID: 5, Mapping: gp.Mapping[1], Address: 0xcafe0_5000},
{ID: 6, Mapping: gp.Mapping[1], Address: 0xcafe0_6000},
}
gp.Sample = []*gprofile.Sample{
{
Location: []*gprofile.Location{gp.Location[0], gp.Location[1]}, // foo ; bar
Value: []int64{1_000_000_000},
},
{
Location: []*gprofile.Location{gp.Location[2], gp.Location[3]}, // 0xcafe0_3000 ; 0xcafe0_4000
Value: []int64{1_000_000_000},
},
{
Location: []*gprofile.Location{gp.Location[4], gp.Location[5]}, // 0xcafe0_5000 ; 0xcafe0_6000
Value: []int64{1_000_000_000},
},
}
gp.SampleType = []*gprofile.ValueType{
{
Type: "cpu",
Unit: "nanoseconds",
},
}
```

When we query a tree back, the symbolizer converts the query to pprof type and with default 8192 max nodes receives the following
```
PeriodType:
Period: 1000000000
Time: 2026-01-26 17:27:43.357 +0700 +07
Samples:
cpu/nanoseconds[dflt]
1000000000: 1 2
2000000000: 3
Locations
1: 0x0 M=1 foo :0:0 s=0()
2: 0x0 M=1 bar :0:0 s=0()
3: 0x0 M=1 other :0:0 s=0
Mappings
1: 0x0/0x0/0x0
```

And If I query pprof with maxNodes=0 (hitting pprof export button) I get a proper profile back.

```
PeriodType:
Period: 1000000000
Time: 2026-01-26 17:31:51.29 +0700 +07
Samples:
cpu/nanoseconds[dflt]
1000000000: 1 2
1000000000: 3 4
1000000000: 5 6
Locations
1: 0x0 M=1 foo :0:0 s=0()
2: 0x0 M=1 bar :0:0 s=0()
3: 0xcafe05000 M=2
4: 0xcafe06000 M=2
5: 0xcafe03000 M=2
6: 0xcafe04000 M=2
Mappings
1: 0x0/0x0/0x0
2: 0x0/0x0/0x0 library.so
```

From a quick look I think the truncation happens here
https://github.com/grafana/pyroscope/blob/35094213f1bfbb8ac0659168caa96bf0d03d845d/pkg/phlaredb/symdb/resolver_pprof_tree.go#L70

If a stacktrace is completely unsymbolized the samples are all merged together. If a stacktrace is partially symbolized - the tree is likely built only on the symbolized frames which is also incorrect.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.