influxdata / influxdata/telegraf

inputs.zookeeper: "unexpected line in mntr response" on ZooKeeper 3.6.3 sketch/summary metrics with quantile labels

Open
#19,488 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[[inputs.zookeeper]]
servers = [":2181"]
```

### Logs from Telegraf

```text
2026-08-18T07:31:24Z I! Starting Telegraf 1.39.3 brought to you by InfluxData the makers of InfluxDB
2026-08-18T07:31:24Z I! Available plugins: 245 inputs, 9 aggregators, 35 processors, 26 parsers, 68 outputs, 8 secret stores
2026-08-18T07:31:24Z I! Loaded inputs: zookeeper
2026-08-18T07:31:24Z I! Loaded aggregators:
2026-08-18T07:31:24Z I! Loaded processors:
2026-08-18T07:31:24Z I! Loaded secretstores:
2026-08-18T07:31:24Z W! Outputs are not used in testing mode!
2026-08-18T07:31:24Z I! Tags enabled: host=crm-prod-zookeeper-1
2026-08-18T07:31:24Z D! [agent] Initializing plugins
2026-08-18T07:31:24Z D! [agent] Starting service inputs
2026-08-18T07:31:24Z E! [inputs.zookeeper] Error in plugin: unexpected line in mntr response: "zk_om_commit_process_time_ms{quantile=\"0.5\"}\tNaN"
2026-08-18T07:31:24Z D! [agent] Stopping service inputs
```

### System info

- Telegraf 1.39.3 (git: HEAD@eb37a442) - OS: Debian GNU/Linux 12 (bookworm) - ZooKeeper: 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT

### Docker

_No response_

### Steps to reproduce

1. Run ZooKeeper 3.6.3 (or later — the sketch/summary metrics were added as part of ZOOKEEPER-3969 and are present in the `mntr` four-letter-word output alongside the plain key/value lines).
2. Configure Telegraf with `[[inputs.zookeeper]]` pointed at that server's client port (default 2181).
3. Run `telegraf --config --test --debug`.

Raw `mntr` output contains, among the normal `zk_key\tvalue` lines, quantile-labeled summary lines such as:
```
zk_om_commit_process_time_ms{quantile="0.5"} NaN
zk_om_commit_process_time_ms{quantile="0.9"} NaN
zk_om_commit_process_time_ms{quantile="0.99"} NaN
zk_om_commit_process_time_ms_count 0.0
zk_om_commit_process_time_ms_sum 0.0
zk_proposal_latency{quantile="0.5"} 1.0
zk_proposal_latency{quantile="0.9"} 3.0
zk_proposal_latency{quantile="0.99"} 66.0
```

### Expected behavior

The plugin either skips lines it doesn't recognize (logging a warning) and continues parsing the rest of the `mntr` response, or supports the quantile-labeled summary line format (matching how ZooKeeper's own `PrometheusMetricsProvider` on the built-in Prometheus endpoint already exposes these same metrics as summaries with a `quantile` label).

### Actual behavior

`plugins/inputs/zookeeper/zookeeper.go`'s line parser uses a strict regex:
```go
var zookeeperFormatRE = regexp.MustCompile(`^zk_(\w[\w\.\-]*)\s+([\w\.\-]+)`)
```
which requires whitespace immediately after the key. Quantile-labeled lines have `{quantile="0.5"}` immediately following the key with no whitespace, so the regex never matches, `FindStringSubmatch` returns fewer than 3 groups, and `gatherServer` returns immediately on the first such line:
```go
parts := zookeeperFormatRE.FindStringSubmatch(line)
if len(parts) != 3 {
return fmt.Errorf("unexpected line in mntr response: %q", line)
}
```
This aborts the entire `mntr` response for that server — none of the metrics from that collection cycle are emitted, not just the offending line.

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in plugins/inputs/zookeeper/zookeeper.go, reading zookeeperFormatRE and gatherServer, then reproduce with telegraf --config --test --debug against ZooKeeper 3.6.3. Done means quantile-labeled mntr lines no longer abort the response: they are either supported or skipped with a warning, while the remaining metrics are emitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.