grafana / grafana/pyroscope

[bug][pyroscope] Tag resolution doesn't appear to be accurate in profiles submitted to /pyroscope/ingest for ruby

Open
#2,072 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Describe the bug

We see callchains that obviously have the wrong stack associated with them when we filter profiles by tags

#### To Reproduce

Use latest weekly build [weekly-f8-1e364093](https://hub.docker.com/layers/grafana/phlare/weekly-f8-1e364093/images/sha256-b8c99494fa8fe73d9499bacd4288cee4583e11be765beafc842b5761702256fc?context=explore) which implements `/pyroscope/ingest`

I wrote is a simple server HTTP server used to intercept the pyroscope payload and save it for analysis, which I also used in https://github.com/grafana/pyroscope/issues/2073:

```
package main

import (
"fmt"
"io"
"os"
"log"
"net/http"
)

var n = 0

func main() {
// Set routing rules
http.HandleFunc("/", Tmp)

//Use the default DefaultServeMux.
err := http.ListenAndServe(":4040", nil)
if err != nil {
log.Fatal(err)
}
}

func Tmp(w http.ResponseWriter, r *http.Request) {
fmt.Printf("request %d %+v\n", n, r)
f, _ := os.Create(fmt.Sprintf("profile%d.pb.gz", n))
defer f.Close()
io.Copy(f, r.Body)
defer r.Body.Close()
n++
}
```

Next we'll use the pyroscope simple ruby app:

https://github.com/grafana/pyroscope/tree/main/examples/ruby/simple

We'll override the server address to point to the interceptor server.

Here is one such captured profile:

[foo.pb.gz](https://github.com/grafana/phlare/files/11165596/foo.pb.gz)

We can see a breakdown of the tags in the profile:

```
go tool pprof -tags ../foo.pb.gz
Main binary filename not available.
function: Total 619.0
485.0 (78.35%): slow
134.0 (21.65%): fast

hostname: Total 619.0
619.0 ( 100%): test

region: Total 619.0
619.0 ( 100%): us-east
```

We'll submit this along to Phlare as so:

```
curl --data-binary @./foo.pb.gz http://127.0.0.1:4100/pyroscope/ingest\?name\=foo&from\=1680742794\&until\=1680742804\&spyName\=rbspy\&sampleRate\=100\&format\=pprof
```

When we view the resulting profile in the Phlare UI, we get this when we filter with `{ pyroscope_app="foo", function="fast"}`:

Screenshot 2023-04-05 at 10 05 19 PM

We shouldn't expect to see the frame `slow_function` here, as the tags should be mutually exclusive:

```
def fast_function
Pyroscope.tag_wrapper({ "function" => "fast" }) do
work(20000)
end
end

def slow_function
Pyroscope.tag_wrapper({ "function" => "slow" }) do
work(80000)
end
end

while true
fast_function
slow_function
end
```

And vice versa, if we query `{ pyroscope_app="foo", function="slow"}`:

Screenshot 2023-04-05 at 10 08 40 PM

#### Expected behavior

The scope of the tags should be accurate. Specifically:

- We should never expect to see samples with callchains containing `slow_function` tagged as `function: fast`
- We should never expect to see samples with callchains containing `fast_function` tagged as `function: slow`

#### Environment

- Infrastructure: Kubernetes
- Deployment tool: jsonnet + custom tooling

#### Additional Context

I originally suspected that this was happening here:

https://github.com/grafana/phlare/blob/4639eee074662648e4e0c9e311bb0ad9aab516cf/pkg/ingester/pyroscope/ingest_adapter.go#L104-L112

The tag is applied to the entire series, but I'm not sure if `pi *storage.PutInput` is scoped only to the particular tag, I would guess that it is not? How is the "tag set" determined?

It could also be that the problem is a bug in the ruby collector, as I notice that the output from `go tool pprof -tags` matches the sample counts we see exactly, suggesting that it could be a source data error and not a bug in the ingestion endpoint, ie:

- What if 14/485 of the source samples tagged as "slow" but actually for "fast_function" were misattributed by the pyroscope ruby gem?
- Likewise, 13/134 of the source samples tagged as "fast" but actually for "slow_function"

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.