Alloy storage and mapping PID errors
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Describe the bug
After deploying pyroscope in micro services mode and using an alloy daemonset for eBPF profiling, everything was working and we were able to see cpu profiles. However, after some time, we are now seeing many errors from the alloy daemonset. If we redeploy alloy, they error stops but eventually comes back.
apologies for the lack of troubleshooting, not really sure where to start.
ex errors
```
Failed to handle mapping for PID 3324373, file /usr/lib64/python3.11/lib-dynload/_bisect.cpython-311-x86_64-linux-gnu.so: failed to update pid_page_to_mapping_info (pid: 3324373, page: 0x7f9e9783b000/52): update: no space left on device
```
```
failed to delete dummy prefix for PID 3324319: batch delete: key does not exist
```
#### To Reproduce
Steps to reproduce the behavior:
1. deploy pyroscope in micro services mode
2. deploy alloy daemon set with the below config
3. wait some amount of time
#### Expected behavior
no error logs without having to redeploy the alloy daemonset
#### Environment
- Infrastructure: Kubernetes on EKS
- Deployment tool: Helm (chart version v1.17.0)
#### Additional Context
alloy config:
```
discovery.kubernetes "all_pods" {
selectors {
field = "spec.nodeName=" + env("HOSTNAME")
role = "pod"
}
role = "pod"
}
discovery.relabel "local_pods" {
targets = discovery.kubernetes.all_pods.targets
rule {
action = "drop"
regex = "Succeeded|Failed"
source_labels = ["__meta_kubernetes_pod_phase"]
}
rule {
action = "replace"
regex = "(.*)@(.*)"
replacement = "ebpf/${1}/${2}"
separator = "@"
source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
target_label = "service_name"
}
rule {
action = "labelmap"
regex = "__meta_kubernetes_pod_label_(.+)"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_pod_node_name"]
target_label = "node"
}
rule {
action = "replace"
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
}
pyroscope.ebpf "instance" {
forward_to = [pyroscope.write.endpoint.receiver]
targets = discovery.relabel.local_pods.output
}
pyroscope.write "endpoint" {
endpoint {
url = "${pyroscope_endpoint}"
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with the supplied Kubernetes discovery, relabel, pyroscope.ebpf, and pyroscope.write configuration, then monitor the Alloy daemonset for the mapping and dummy-prefix errors. Compare behavior before and after redeploying Alloy and investigate the reported no-space-left-on-device condition; done means the errors do not return during sustained operation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kubernetes, linux, python
- Domain
- devops, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100