[Workload]: metrics-emitter

Workload Name

metrics-emitter

Workload Description

A workload that runs a Prometheus-compatible metrics endpoint inside a VM, exposing configurable gauges, counters, and histograms with tunable label cardinality and scrape-ready /metrics path. Produces sustained, realistic metric signals that monitoring partners can scrape via OpenShift's native ServiceMonitor/PodMonitor pipeline.

This is the missing link between virtwork's in-VM workloads and the Prometheus-based monitoring ecosystem on OpenShift. The current workloads (cpu, memory, disk, database) generate real system pressure, but monitoring partners need scrapeable application metrics — not just node-level signals. A metrics-emitter VM acts as a synthetic application that exposes the same kind of metric surface a real business application would: request duration histograms, active connection gauges, error counters, queue depth gauges — all with configurable cardinality to stress the partner's ingestion pipeline.

Tooling and Packages

• Tool: avalanche (Prometheus metric load generator by Prometheus team) or a lightweight custom exporter
• avalanche flags: --metric-count=500 --label-count=10 --series-count=5 --port=9101 --value-interval=30
• RPM packages: none — avalanche is a single Go binary
• systemd service command: /usr/local/bin/avalanche --metric-count=500 --label-count=10 --series-count=5 --port=9101
• Exposes port 9101 (Prometheus metrics) via Kubernetes Service
• Configurable parameters: metric count, label count (cardinality), series count per metric, value change interval, port

VM Count Model

Single VM (like cpu, memory, disk)

Required Resources

• Kubernetes Service (for inter-VM communication)
• Persistent storage (DataVolume)
• Kubernetes Secret (for credentials or config)
• Additional CPU/memory beyond defaults
• GPU or special device passthrough

Cloud-Init Details

write_files:
  - path: /etc/systemd/system/virtwork-metrics-emitter.service
    content: |
      [Unit]
      Description=Virtwork Prometheus metrics emitter
      After=network-online.target
      Wants=network-online.target
      [Service]
      Type=simple
      ExecStart=/usr/local/bin/avalanche \
        --metric-count=500 \
        --label-count=10 \
        --series-count=5 \
        --port=9101 \
        --value-interval=30
      Restart=always
      RestartSec=5
      [Install]
      WantedBy=multi-user.target
runcmd:
  - curl -Lo /usr/local/bin/avalanche https://github.com/prometheus-community/avalanche/releases/download/v0.6.0/avalanche_0.6.0_linux_amd64.tar.gz
  - tar -xzf /usr/local/bin/avalanche -C /usr/local/bin/
  - chmod +x /usr/local/bin/avalanche
  - systemctl enable --now virtwork-metrics-emitter.service

The Kubernetes Service should expose port 9101 with appropriate labels for ServiceMonitor discovery:

apiVersion: v1
kind: Service
metadata:
  name: virtwork-metrics-emitter-0
  labels:
    app.kubernetes.io/managed-by: virtwork
    virtwork/component: metrics-emitter
    virtwork/metrics: "true"
spec:
  ports:
    - name: metrics
      port: 9101
      targetPort: 9101
  selector:
    # matches VM pod labels

Use Case

• Monitoring platform partners (Datadog, Dynatrace, Sysdig, Grafana Labs): Need scrapeable Prometheus endpoints from VMs to validate that their agents correctly discover, scrape, and ingest application-level metrics from KubeVirt VMs — not just node/kubelet metrics. High cardinality (many labels × many series) stresses ingestion pipelines and validates the partner's ability to handle metric volume at scale.
• Prometheus/Thanos/Mimir ecosystem partners: Need realistic scrape targets to validate long-term storage, compaction, and query performance against VM-sourced metrics. Configurable metric count and cardinality lets partners tune the load to match their customers' real-world profiles.
• OpenShift platform validation: Validates that the full Prometheus pipeline — ServiceMonitor discovery → scrape → remote write → partner product — works correctly when the scrape target is a KubeVirt VM rather than a pod. This exercises a code path that's distinct from container-based scraping and is less commonly tested.
• Capacity planning partners: Need predictable, tunable metric volume to validate alerting thresholds and dashboard rendering under load.

Additional Context

• avalanche is maintained by the Prometheus community and designed specifically for this purpose: generating configurable Prometheus metric load. It's the standard tool for testing Prometheus-compatible backends.
• The Kubernetes Service is critical — it enables OpenShift's ServiceMonitor to discover and scrape the VM's metrics endpoint through the standard Prometheus operator pipeline. This is what makes this workload uniquely valuable on OpenShift vs. a bare VM.
• Configurable parameters to expose via CLI/YAML: metric-count (number of distinct metrics), label-count (labels per metric — controls cardinality), series-count (series per metric), value-interval (seconds between value changes), port.
• Consider also generating a sample ServiceMonitor CR in dry-run output to show partners how to wire up scraping. This isn't a virtwork-managed resource but a documentation/example artifact.
• At 500 metrics × 5 series × 10 labels, a single VM produces ~2,500 time series — comparable to a moderately instrumented microservice. Scaling to 5,000+ metrics or 50+ labels tests high-cardinality ingestion.

Start by locating the existing cpu, memory, and disk workload implementations and their Kubernetes Service generation. Review how cloud-init and systemd services are defined, then compare the proposed avalanche command and port 9101 Service labels with the workload conventions. Done means a metrics-emitter VM exposes the configured Prometheus endpoint and its Service supports ServiceMonitor or PodMonitor discovery.