grafana / grafana/pyroscope

Alloy storage and mapping PID errors

Open
#4,862 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Describe the bug
After deploying pyroscope in micro services mode and using an alloy daemonset for eBPF profiling, everything was working and we were able to see cpu profiles. However, after some time, we are now seeing many errors from the alloy daemonset. If we redeploy alloy, they error stops but eventually comes back.
apologies for the lack of troubleshooting, not really sure where to start.

ex errors
```
Failed to handle mapping for PID 3324373, file /usr/lib64/python3.11/lib-dynload/_bisect.cpython-311-x86_64-linux-gnu.so: failed to update pid_page_to_mapping_info (pid: 3324373, page: 0x7f9e9783b000/52): update: no space left on device
```
```
failed to delete dummy prefix for PID 3324319: batch delete: key does not exist
```

#### To Reproduce
Steps to reproduce the behavior:

1. deploy pyroscope in micro services mode
2. deploy alloy daemon set with the below config
3. wait some amount of time

#### Expected behavior

no error logs without having to redeploy the alloy daemonset

#### Environment

- Infrastructure: Kubernetes on EKS
- Deployment tool: Helm (chart version v1.17.0)

#### Additional Context

alloy config:
```
discovery.kubernetes "all_pods" {
selectors {
field = "spec.nodeName=" + env("HOSTNAME")
role = "pod"
}
role = "pod"
}
discovery.relabel "local_pods" {
targets = discovery.kubernetes.all_pods.targets
rule {
action = "drop"
regex = "Succeeded|Failed"
source_labels = ["__meta_kubernetes_pod_phase"]
}
rule {
action = "replace"
regex = "(.*)@(.*)"
replacement = "ebpf/${1}/${2}"
separator = "@"
source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
target_label = "service_name"
}
rule {
action = "labelmap"
regex = "__meta_kubernetes_pod_label_(.+)"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_pod_node_name"]
target_label = "node"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
}
pyroscope.ebpf "instance" {
forward_to = [pyroscope.write.endpoint.receiver]
targets = discovery.relabel.local_pods.output
}
pyroscope.write "endpoint" {
endpoint {
url = "${pyroscope_endpoint}"
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the supplied Kubernetes discovery, relabel, pyroscope.ebpf, and pyroscope.write configuration, then monitor the Alloy daemonset for the mapping and dummy-prefix errors. Compare behavior before and after redeploying Alloy and investigate the reported no-space-left-on-device condition; done means the errors do not return during sustained operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes, linux, python
Domain
devops, infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.