influxdata / influxdata/telegraf
inputs.zookeeper: "unexpected line in mntr response" on ZooKeeper 3.6.3 sketch/summary metrics with quantile labels
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[[inputs.zookeeper]]
servers = [":2181"]
```
### Logs from Telegraf
```text
2026-08-18T07:31:24Z I! Starting Telegraf 1.39.3 brought to you by InfluxData the makers of InfluxDB
2026-08-18T07:31:24Z I! Available plugins: 245 inputs, 9 aggregators, 35 processors, 26 parsers, 68 outputs, 8 secret stores
2026-08-18T07:31:24Z I! Loaded inputs: zookeeper
2026-08-18T07:31:24Z I! Loaded aggregators:
2026-08-18T07:31:24Z I! Loaded processors:
2026-08-18T07:31:24Z I! Loaded secretstores:
2026-08-18T07:31:24Z W! Outputs are not used in testing mode!
2026-08-18T07:31:24Z I! Tags enabled: host=crm-prod-zookeeper-1
2026-08-18T07:31:24Z D! [agent] Initializing plugins
2026-08-18T07:31:24Z D! [agent] Starting service inputs
2026-08-18T07:31:24Z E! [inputs.zookeeper] Error in plugin: unexpected line in mntr response: "zk_om_commit_process_time_ms{quantile=\"0.5\"}\tNaN"
2026-08-18T07:31:24Z D! [agent] Stopping service inputs
```
### System info
- Telegraf 1.39.3 (git: HEAD@eb37a442) - OS: Debian GNU/Linux 12 (bookworm) - ZooKeeper: 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
### Docker
_No response_
### Steps to reproduce
1. Run ZooKeeper 3.6.3 (or later — the sketch/summary metrics were added as part of ZOOKEEPER-3969 and are present in the `mntr` four-letter-word output alongside the plain key/value lines).
2. Configure Telegraf with `[[inputs.zookeeper]]` pointed at that server's client port (default 2181).
3. Run `telegraf --config --test --debug`.
Raw `mntr` output contains, among the normal `zk_key\tvalue` lines, quantile-labeled summary lines such as:
```
zk_om_commit_process_time_ms{quantile="0.5"} NaN
zk_om_commit_process_time_ms{quantile="0.9"} NaN
zk_om_commit_process_time_ms{quantile="0.99"} NaN
zk_om_commit_process_time_ms_count 0.0
zk_om_commit_process_time_ms_sum 0.0
zk_proposal_latency{quantile="0.5"} 1.0
zk_proposal_latency{quantile="0.9"} 3.0
zk_proposal_latency{quantile="0.99"} 66.0
```
### Expected behavior
The plugin either skips lines it doesn't recognize (logging a warning) and continues parsing the rest of the `mntr` response, or supports the quantile-labeled summary line format (matching how ZooKeeper's own `PrometheusMetricsProvider` on the built-in Prometheus endpoint already exposes these same metrics as summaries with a `quantile` label).
### Actual behavior
`plugins/inputs/zookeeper/zookeeper.go`'s line parser uses a strict regex:
```go
var zookeeperFormatRE = regexp.MustCompile(`^zk_(\w[\w\.\-]*)\s+([\w\.\-]+)`)
```
which requires whitespace immediately after the key. Quantile-labeled lines have `{quantile="0.5"}` immediately following the key with no whitespace, so the regex never matches, `FindStringSubmatch` returns fewer than 3 groups, and `gatherServer` returns immediately on the first such line:
```go
parts := zookeeperFormatRE.FindStringSubmatch(line)
if len(parts) != 3 {
return fmt.Errorf("unexpected line in mntr response: %q", line)
}
```
This aborts the entire `mntr` response for that server — none of the metrics from that collection cycle are emitted, not just the offending line.
### Additional info
_No response_
Contributor guide
Research direction
Start in plugins/inputs/zookeeper/zookeeper.go, reading zookeeperFormatRE and gatherServer, then reproduce with telegraf --config --test --debug against ZooKeeper 3.6.3. Done means quantile-labeled mntr lines no longer abort the response: they are either supported or skipped with a warning, while the remaining metrics are emitted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100