hyperdxio / hyperdxio/hyperdx

[Bug] CPU and Events data missing on Kubernetes Dashboard

Open
#1,214 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
9.9k
Forks
471
Avg merge
2d 4h
Merged PRs (30d)
117

Description

Hello! We've just deployed Clickstack on a brand new bare metal K8s cluster. We are using the helm chart with the hyperdx and Otel enabled (we already have an external ClickHouse).

After the deployment of the chart, and installed the cluster and node level Otel collectors as per documentation, the data seems to be flowing. However, we have a few weird things which seems to me something is wrong:

  1. In all the Kubernetes dashboard tabs (pods, nodes and namespace), the "CPU Usage" charts are empty (Memory Usage works fine).
Image Image Image
  1. Also, in the second picture you may see that all 3 nodes have Status "Not Ready", CPU N/A, when clearly the nodes are healthy and working.
  2. On the Pod details and on the Pods tab, they show no events, when we clearly have events as shown by kubectl and on other tools like Lens.
Image Image

Is there anything we are missing for configuring the kubernetes data besides what was mentioned in the docs?

Here are the settings applied to each helm chart:

  • clickstack:
clickhouse:
  enabled: false
global:
  keepPVC: true
  storageClassName: data-block-sc
hyperdx:
  env:
  - name: NODE_EXTRA_CA_CERTS
    value: /etc/ssl/certs/ca-certificates.crt
  existingConfigSecret: clickstack-hyperdx-config
  frontendUrl: https://<redacted>
  image:
    tag: 2.5.0
  podDisruptionBudget:
    enabled: true
  replicas: 3
  useExistingConfigSecret: true
mongodb:
  enabled: true
  image: mongo:8.0.14-noble
  persistence:
    dataSize: 20Gi
    enabled: true
    storageClass: data-block-sc
  port: 27017
otel:
  clickhouseDatabase: clickstack
  clickhouseEndpoint: clickhouse://<host>.clickhouse.svc.cluster.local:9440?secure=true
  clickhousePassword: <redacted>
  clickhouseUser: <redacted>
  enabled: true
  image:
    tag: 2.5.0
  replicas: 3

Otel collector - cluster deployment:

config:
  exporters:
    otlphttp:
      compression: gzip
      endpoint: http://clickstack-hdx-oss-v2-otel-collector:4318
      headers:
        authorization: ${env:API_KEY}
  service:
    pipelines:
      logs:
        exporters:
        - otlphttp
      metrics:
        exporters:
        - otlphttp
extraEnvs:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      key: API_KEY
      name: ingestion-api-key
      optional: true
image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib
  tag: 0.136.0
mode: deployment
presets:
  clusterMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
    extractAllPodAnnotations: true
    extractAllPodLabels: true
  kubernetesEvents:
    enabled: true
replicaCount: 1

Otel collector - Node daemonset:

clusterRole:
  create: true
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes/proxy
    - pods
    - nodes
    - namespaces
    verbs:
    - get
    - list
    - watch
config:
  exporters:
    otlphttp:
      compression: gzip
      endpoint: http://clickstack-hdx-oss-v2-otel-collector:4318
      headers:
        authorization: ${env:API_KEY}
  receivers:
    kubeletstats:
      auth_type: serviceAccount
      collection_interval: 20s
      endpoint: ${env:K8S_NODE_NAME}:10250
      insecure_skip_verify: true
      metrics:
        container.uptime:
          enabled: true
        k8s.container.cpu_limit_utilization:
          enabled: true
        k8s.container.cpu_request_utilization:
          enabled: true
        k8s.container.memory_limit_utilization:
          enabled: true
        k8s.container.memory_request_utilization:
          enabled: true
        k8s.node.uptime:
          enabled: true
        k8s.pod.cpu_limit_utilization:
          enabled: true
        k8s.pod.cpu_request_utilization:
          enabled: true
        k8s.pod.memory_limit_utilization:
          enabled: true
        k8s.pod.memory_request_utilization:
          enabled: true
        k8s.pod.uptime:
          enabled: true
  service:
    pipelines:
      logs:
        exporters:
        - otlphttp
      metrics:
        exporters:
        - otlphttp
extraEnvs:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      key: API_KEY
      name: ingestion-api-key
      optional: true
image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib
  tag: 0.136.0
mode: daemonset
presets:
  hostMetrics:
    enabled: true
  kubeletMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
    extractAllPodAnnotations: true
    extractAllPodLabels: true
  logsCollection:
    enabled: true

Any help would be appreciated.

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the empty CPU charts, Not Ready node status, and missing pod events in the Kubernetes dashboard, then compare the configured kubeletstats, clusterMetrics, and kubernetesEvents collector pipelines with the data received by ClickHouse. Done means CPU and node status metrics and Kubernetes events appear in the relevant dashboard tabs.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, helm, kubernetes
Domain
devops, infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.