FunctionsStatsGenerator - Failed to collect metrics
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
While doing some testing on master (2.8.0-SNAPSHOT), this message started flooding my broker logs as soon as I started the brokers:
```
Apr 30 21:25:40 fab08 77c203532c67: 2021-05-01T03:25:40,933 [prometheus-stats-47-1] WARN org.apache.pulsar.functions.worker.FunctionsStatsGenerator - Failed to collect metrics for function instance myTenant/myNamespace/function-router:0
Apr 30 21:25:40 fab08 77c203532c67: java.net.ConnectException: Connection refused (Connection refused)
Apr 30 21:25:40 fab08 77c203532c67: #011at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.net.Socket.connect(Socket.java:556) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.NetworkClient.doConnect(NetworkClient.java:180) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.http.HttpClient.(HttpClient.java:242) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) ~[?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.pulsar.functions.runtime.RuntimeUtils.getPrometheusMetrics(RuntimeUtils.java:447) ~[org.apache.pulsar-pulsar-functions-runtime-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.pulsar.functions.runtime.process.ProcessRuntime.getPrometheusMetrics(ProcessRuntime.java:321) ~[org.apache.pulsar-pulsar-functions-runtime-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.pulsar.functions.worker.FunctionsStatsGenerator.generate(FunctionsStatsGenerator.java:71) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.pulsar.functions.worker.PulsarWorkerService.generateFunctionsStats(PulsarWorkerService.java:171) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:106) ~[org.apache.pulsar-pulsar-broker-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:72) ~[org.apache.pulsar-pulsar-broker-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Apr 30 21:25:40 fab08 77c203532c67: #011at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.13.0.jar:4.13.0]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
Apr 30 21:25:40 fab08 77c203532c67: #011at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
Apr 30 21:25:40 fab08 77c203532c67: #011at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
```
That function was created in a previous version (Pulsar 2.4.0), and the brokers were upgraded to master.
Contributor guide
Research direction
Start with FunctionsStatsGenerator.java at generate(), then follow the reported path through ProcessRuntime.java and RuntimeUtils.java at line 447. Reproduce the metrics collection failure for a function created on Pulsar 2.4.0 after upgrading to master, and determine the expected handling of the refused connection. Done should address the repeated warning reported during broker startup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100