[Bug] CPU and Events data missing on Kubernetes Dashboard
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 9.9k
- Forks
- 471
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 117
Description
Hello! We've just deployed Clickstack on a brand new bare metal K8s cluster. We are using the helm chart with the hyperdx and Otel enabled (we already have an external ClickHouse).
After the deployment of the chart, and installed the cluster and node level Otel collectors as per documentation, the data seems to be flowing. However, we have a few weird things which seems to me something is wrong:
- In all the Kubernetes dashboard tabs (pods, nodes and namespace), the "CPU Usage" charts are empty (Memory Usage works fine).
- Also, in the second picture you may see that all 3 nodes have Status "Not Ready", CPU N/A, when clearly the nodes are healthy and working.
- On the Pod details and on the Pods tab, they show no events, when we clearly have events as shown by kubectl and on other tools like Lens.
Is there anything we are missing for configuring the kubernetes data besides what was mentioned in the docs?
Here are the settings applied to each helm chart:
- clickstack:
clickhouse:
enabled: false
global:
keepPVC: true
storageClassName: data-block-sc
hyperdx:
env:
- name: NODE_EXTRA_CA_CERTS
value: /etc/ssl/certs/ca-certificates.crt
existingConfigSecret: clickstack-hyperdx-config
frontendUrl: https://<redacted>
image:
tag: 2.5.0
podDisruptionBudget:
enabled: true
replicas: 3
useExistingConfigSecret: true
mongodb:
enabled: true
image: mongo:8.0.14-noble
persistence:
dataSize: 20Gi
enabled: true
storageClass: data-block-sc
port: 27017
otel:
clickhouseDatabase: clickstack
clickhouseEndpoint: clickhouse://<host>.clickhouse.svc.cluster.local:9440?secure=true
clickhousePassword: <redacted>
clickhouseUser: <redacted>
enabled: true
image:
tag: 2.5.0
replicas: 3
Otel collector - cluster deployment:
config:
exporters:
otlphttp:
compression: gzip
endpoint: http://clickstack-hdx-oss-v2-otel-collector:4318
headers:
authorization: ${env:API_KEY}
service:
pipelines:
logs:
exporters:
- otlphttp
metrics:
exporters:
- otlphttp
extraEnvs:
- name: API_KEY
valueFrom:
secretKeyRef:
key: API_KEY
name: ingestion-api-key
optional: true
image:
repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib
tag: 0.136.0
mode: deployment
presets:
clusterMetrics:
enabled: true
kubernetesAttributes:
enabled: true
extractAllPodAnnotations: true
extractAllPodLabels: true
kubernetesEvents:
enabled: true
replicaCount: 1
Otel collector - Node daemonset:
clusterRole:
create: true
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- pods
- nodes
- namespaces
verbs:
- get
- list
- watch
config:
exporters:
otlphttp:
compression: gzip
endpoint: http://clickstack-hdx-oss-v2-otel-collector:4318
headers:
authorization: ${env:API_KEY}
receivers:
kubeletstats:
auth_type: serviceAccount
collection_interval: 20s
endpoint: ${env:K8S_NODE_NAME}:10250
insecure_skip_verify: true
metrics:
container.uptime:
enabled: true
k8s.container.cpu_limit_utilization:
enabled: true
k8s.container.cpu_request_utilization:
enabled: true
k8s.container.memory_limit_utilization:
enabled: true
k8s.container.memory_request_utilization:
enabled: true
k8s.node.uptime:
enabled: true
k8s.pod.cpu_limit_utilization:
enabled: true
k8s.pod.cpu_request_utilization:
enabled: true
k8s.pod.memory_limit_utilization:
enabled: true
k8s.pod.memory_request_utilization:
enabled: true
k8s.pod.uptime:
enabled: true
service:
pipelines:
logs:
exporters:
- otlphttp
metrics:
exporters:
- otlphttp
extraEnvs:
- name: API_KEY
valueFrom:
secretKeyRef:
key: API_KEY
name: ingestion-api-key
optional: true
image:
repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib
tag: 0.136.0
mode: daemonset
presets:
hostMetrics:
enabled: true
kubeletMetrics:
enabled: true
kubernetesAttributes:
enabled: true
extractAllPodAnnotations: true
extractAllPodLabels: true
logsCollection:
enabled: true
Any help would be appreciated.
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the empty CPU charts, Not Ready node status, and missing pod events in the Kubernetes dashboard, then compare the configured kubeletstats, clusterMetrics, and kubernetesEvents collector pipelines with the data received by ClickHouse. Done means CPU and node status metrics and Kubernetes events appear in the relevant dashboard tabs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, helm, kubernetes
- Domain
- devops, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100