JMX to Prometheus errors

Hi, I'm on docker setup.
I'm trying to implement JMX to Prometheus https://docs.pinot.apache.org/operators/operating-pinot/monitoring

This is my docker-compose

version: '3.7'
services:
zookeeper:
image: zookeeper:3.5.6
container_name: "zookeeper-wiki"
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: wurstmeister/kafka:latest
restart: unless-stopped
container_name: "kafka-wiki"
ports:
- "9092:9092"
expose:
- "9093"
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper-wiki:2181/kafka
KAFKA_BROKER_ID: 0
KAFKA_ADVERTISED_HOST_NAME: kafka-wiki
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-wiki:9093,OUTSIDE://localhost:9092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
pinot-controller:
image: apachepinot/pinot:0.12.0
command: "StartController -zkAddress zookeeper-wiki:2181 -dataDir /data"
container_name: "pinot-controller-wiki"
volumes:
- ./config:/config
- ./data:/data
restart: unless-stopped
ports:
- "9000:9000"
- "8079:8079"
environment:
- _JAVA_OPTIONS="-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8079:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms4G -Xmx4G -XX:MaxDirectMemorySize=30g -Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml -Dplugins.dir=$BASEDIR/plugins"
depends_on:
- zookeeper
pinot-broker:
image: apachepinot/pinot:0.12.0
command: "StartBroker -zkAddress zookeeper-wiki:2181"
restart: unless-stopped
container_name: "pinot-broker-wiki"
volumes:
- ./config:/config
ports:
- "8099:8099"
- "8081:8081"
environment:
- _JAVA_OPTIONS="-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8081:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms4G -Xmx4G -XX:MaxDirectMemorySize=30g -Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml -Dplugins.dir=$BASEDIR/plugins"
depends_on:
- pinot-controller
pinot-server:
image: apachepinot/pinot:0.12.0
command: "StartServer -zkAddress zookeeper-wiki:2181"
restart: unless-stopped
container_name: "pinot-server-wiki"
volumes:
- ./config:/config
ports:
- "8098:8098"
- "8082:8082"
environment:
- _JAVA_OPTIONS="-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8082:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms4G -Xmx4G -XX:MaxDirectMemorySize=30g -Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml -Dplugins.dir=$BASEDIR/plugins"
depends_on:
- pinot-broker

I'm not able to see the metrics on prometheus. Since I'm on docker, should I change anything?

I'm getting this on terminal : pinot-server-wiki | bool UseG1GC = true
{product} {ergonomic}
pinot-server-wiki |
pinot-server-wiki | Logging:
pinot-server-wiki | Log output configuration:
pinot-server-wiki | #0: stdout all=warning uptime,level,tags
pinot-server-wiki | #1: stderr all=off uptime,level,tags
pinot-server-wiki |
pinot-server-wiki | Environment Variables:
pinot-server-wiki | JAVA_HOME=/usr/local/openjdk-11
pinot-server-wiki | _JAVA_OPTIONS="-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8082:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms4G -Xmx4G
-XX:MaxDirectMemorySize=30g -Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml -Dplugins.dir=/plugins"
pinot-server-wiki | PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
pinot-server-wiki | LANG=C.UTF-8
pinot-server-wiki |
pinot-server-wiki | Signal Handlers:
pinot-server-wiki | SIGSEGV: [libjvm.so+0xd9b640], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGBUS: [libjvm.so+0xd9b640], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGFPE: [libjvm.so+0xd9b640], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGPIPE: [libjvm.so+0xb9de10], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGXFSZ: [libjvm.so+0xb9de10], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGILL: [libjvm.so+0xd9b640], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGUSR2: [libjvm.so+0xb9dca0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGHUP: [libjvm.so+0xb9e460], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGINT: [libjvm.so+0xb9e460], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGTERM: [libjvm.so+0xb9e460], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki | SIGQUIT: [libjvm.so+0xb9e460], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
pinot-server-wiki |
pinot-server-wiki |
pinot-server-wiki | --------------- S Y S T E M ---------------
pinot-server-wiki |
pinot-server-wiki | OS:PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
pinot-server-wiki | NAME="Debian GNU/Linux"
pinot-server-wiki | VERSION_ID="11"
pinot-server-wiki | VERSION="11 (bullseye)"
pinot-server-wiki | VERSION_CODENAME=bullseye
pinot-server-wiki | ID=debian
pinot-server-wiki | HOME_URL="https://www.debian.org/"
pinot-server-wiki | SUPPORT_URL="https://www.debian.org/support"
pinot-server-wiki | BUG_REPORT_URL="https://bugs.debian.org/"
pinot-server-wiki | uname:Linux 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC
2023 x86_64
pinot-server-wiki | OS uptime: 0 days 0:41 hours
pinot-server-wiki | libc:glibc 2.31 NPTL 2.31
pinot-server-wiki | rlimit (soft/hard): STACK 8192k/infinity , CORE 0k/infinity , NPROC infinity/infinity , NOFILE 1048576/1048576 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK 82000k/82000k
pinot-server-wiki | load average:0.32 0.74 0.97
pinot-server-wiki |
pinot-server-wiki | /proc/meminfo:
pinot-server-wiki | MemTotal: 3996436 kB
pinot-server-wiki | MemFree: 1986152 kB
pinot-server-wiki | MemAvailable: 2581700 kB
pinot-server-wiki | Buffers: 51496 kB
pinot-server-wiki | Cached: 729564 kB
pinot-server-wiki | SwapCached: 17916 kB
pinot-server-wiki | Active: 550504 kB
pinot-server-wiki | Inactive: 1114828 kB
pinot-server-wiki | Active(anon): 141712 kB
pinot-server-wiki | Inactive(anon): 747188 kB
pinot-server-wiki | Active(file): 408792 kB
pinot-server-wiki | Inactive(file): 367640 kB
pinot-server-wiki | Unevictable: 0 kB
pinot-server-wiki | Mlocked: 0 kB
pinot-server-wiki | SwapTotal: 1048576 kB
pinot-server-wiki | SwapFree: 730680 kB
pinot-server-wiki | Dirty: 572 kB
pinot-server-wiki | Writeback: 4 kB
pinot-server-wiki | AnonPages: 858944 kB
pinot-server-wiki | Mapped: 197644 kB
pinot-server-wiki | Shmem: 4588 kB
pinot-server-wiki | KReclaimable: 38140 kB
pinot-server-wiki | Slab: 91884 kB
pinot-server-wiki | SReclaimable: 38140 kB
pinot-server-wiki | SUnreclaim: 53744 kB
pinot-server-wiki | KernelStack: 13632 kB
pinot-server-wiki | PageTables: 10020 kB
pinot-server-wiki | NFS_Unstable: 0 kB
pinot-server-wiki | Bounce: 0 kB
pinot-server-wiki | WritebackTmp: 0 kB
pinot-server-wiki | CommitLimit: 3046792 kB
pinot-server-wiki | Committed_AS: 9514316 kB
pinot-server-wiki | VmallocTotal: 34359738367 kB
pinot-server-wiki | VmallocUsed: 34280 kB
pinot-server-wiki | VmallocChunk: 0 kB
pinot-server-wiki | Percpu: 2048 kB
pinot-server-wiki | AnonHugePages: 456704 kB
pinot-server-wiki | ShmemHugePages: 0 kB
pinot-server-wiki | ShmemPmdMapped: 0 kB
pinot-server-wiki | FileHugePages: 0 kB
pinot-server-wiki | FilePmdMapped: 0 kB
pinot-server-wiki | HugePages_Total: 0
pinot-server-wiki | HugePages_Free: 0
pinot-server-wiki | HugePages_Rsvd: 0
pinot-server-wiki | HugePages_Surp: 0
pinot-server-wiki | Hugepagesize: 2048 kB
pinot-server-wiki | Hugetlb: 0 kB
pinot-server-wiki | DirectMap4k: 25600 kB
pinot-server-wiki | DirectMap2M: 3100672 kB
pinot-server-wiki | DirectMap1G: 9437184 kB
pinot-server-wiki |
pinot-server-wiki | /sys/kernel/mm/transparent_hugepage/enabled:
pinot-server-wiki | [always] madvise never
pinot-server-wiki | /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):
pinot-server-wiki | always defer defer+madvise [madvise] never
pinot-server-wiki |
pinot-server-wiki | Process Memory:
pinot-server-wiki | Virtual Size: 6958564K (peak: 15711204K)
pinot-server-wiki | Resident Set Size: 128236K (peak: 128236K) (anon: 102140K, file: 26096K, shmem: 0K)
pinot-server-wiki | Swapped out: 0K
pinot-server-wiki | C-Heap outstanding allocations: 33680K, retained: 4463K
pinot-server-wiki | glibc malloc tunables: (default)
pinot-server-wiki |
pinot-server-wiki | /proc/sys/kernel/threads-max (system-wide limit on the number of threads):
pinot-server-wiki | 31171
pinot-server-wiki | /proc/sys/vm/max_map_count (maximum number of memory map areas a process may
have):
pinot-server-wiki | 65530
pinot-server-wiki | /proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
pinot-server-wiki | 32768
pinot-server-wiki |
pinot-server-wiki | container (cgroup) information:
pinot-server-wiki | container_type: cgroupv1
pinot-server-wiki | cpu_cpuset_cpus: 0-3
pinot-server-wiki | cpu_memory_nodes: 0
pinot-server-wiki | active_processor_count: 4
pinot-server-wiki | cpu_quota: no quota
pinot-server-wiki | cpu_period: 100000
pinot-server-wiki | cpu_shares: no shares
pinot-server-wiki | memory_limit_in_bytes: unlimited
pinot-server-wiki | memory_and_swap_limit_in_bytes: unlimited
pinot-server-wiki | memory_soft_limit_in_bytes: unlimited
pinot-server-wiki | memory_usage_in_bytes: 106950656
pinot-server-wiki | memory_max_usage_in_bytes: 106995712
pinot-server-wiki |
pinot-server-wiki | Hyper-V virtualization detected
pinot-server-wiki | Steal ticks since vm start: 0
pinot-server-wiki | Steal ticks percentage since vm start: 0.000
pinot-server-wiki |
pinot-server-wiki | CPU:total 4 (initial active 4) (2 cores per cpu, 2 threads per core) family 6 model 158 stepping 9 microcode 0xffffffff, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, aes, clmul, erms, 3dnowpref, lzcnt, ht, tsc
pinot-server-wiki | CPU Model and flags from /proc/cpuinfo:
pinot-server-wiki | model name : Intel(R) Pentium(R) CPU G4560 @ 3.50GHz
pinot-server-wiki | flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 cx16 pdcm pcid sse4_1 sse4_2 movbe popcnt aes xsave rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase smep erms invpcid rdseed smap clflushopt xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities
pinot-server-wiki | Online cpus:
pinot-server-wiki | 0-3
pinot-server-wiki | Offline cpus:
pinot-server-wiki |
pinot-server-wiki | BIOS frequency limitation:
pinot-server-wiki |
pinot-server-wiki | Frequency switch latency (ns):
pinot-server-wiki |
pinot-server-wiki | Available cpu frequencies:
pinot-server-wiki |
pinot-server-wiki | Current governor:
pinot-server-wiki |
pinot-server-wiki | Core performance/turbo boost:
pinot-server-wiki |
pinot-server-wiki |
pinot-server-wiki | Memory: 4k page, physical 3996436k(1986152k free), swap 1048576k(730680k free)
pinot-server-wiki |
pinot-server-wiki | vm_info: OpenJDK 64-Bit Server VM (11.0.16+8) for linux-amd64 JRE (11.0.16+8), built on Jul 16 2022 23:58:30 by "openjdk" with gcc 8.3.1 20190311 (Red Hat 8.3.1-3)
pinot-server-wiki |
pinot-server-wiki | END.
pinot-server-wiki | #
pinot-server-wiki | # If you would like to submit a bug report, please visit:
pinot-server-wiki | # https://bugreport.java.com/bugreport/crash.jsp

Begin with the supplied docker-compose services and their _JAVA_OPTIONS values; compare the JMX Prometheus agent paths and exposed ports with the linked Pinot monitoring instructions. Reproduce the startup and log behavior for the controller, broker, and server, then verify the configured metrics endpoints; done means the endpoints expose metrics and Prometheus can scrape them.

Written by the indexing model from the issue text.