influxdata / influxdata/telegraf

MQTT input sent to two different outputs completely stops reporting if one output unresponsive

Open
#16,537 5 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[agent]
debug = true
quiet = false
interval = "10s"

### --- setup for just mocks ---- ###
[[inputs.mock]]
metric_name = "fake_alpha"

[inputs.mock.tags]
destination_topic = "alpha"

[[inputs.mock.constant]]
name = "fake_val"
value = "1234"

[[inputs.mock]]
metric_name = "fake_beta"

[inputs.mock.tags]
destination_topic = "beta"

[[inputs.mock.constant]]
name = "fake_val"
value = "5678"

[[outputs.mqtt]]
client_id = "fake_data_client"
namepass = ["fake*"]
servers = ["mqtt://mosquitto:1883"]
data_format = "json"
topic = "{{ .Tag \"destination_topic\" }}"

### -- end setup for mock data -- ###

[[inputs.mqtt_consumer]]
servers = ["mqtt://mosquitto:1883"]
data_format = "json"
topics = ["alpha"]
name_override = "alpha_mqtt"

[[inputs.mqtt_consumer]]
servers = ["mqtt://mosquitto:1883"]
data_format = "json"
topics = ["beta"]
name_override = "beta_mqtt"
max_undelivered_messages = 10

[[outputs.file]]
files = ["stdout"]
data_format = "json"
namepass = ["alpha_mqtt", "beta_mqtt"]

[[outputs.http]]
url = "http://notarealsite1234.com"
method = "PUT"
data_format = "json"
metric_batch_size = 1
```

### Logs from Telegraf

```text
2025-02-20T18:27:25Z I! Loading config: /etc/telegraf/telegraf.conf
2025-02-20T18:27:25Z I! Starting Telegraf 1.33.2 brought to you by InfluxData the makers of InfluxDB
2025-02-20T18:27:25Z I! Available plugins: 236 inputs, 9 aggregators, 33 processors, 26 parsers, 63 outputs, 6 secret-stores
2025-02-20T18:27:25Z I! Loaded inputs: mock (2x) mqtt_consumer (2x)
2025-02-20T18:27:25Z I! Loaded aggregators:
2025-02-20T18:27:25Z I! Loaded processors:
2025-02-20T18:27:25Z I! Loaded secretstores:
2025-02-20T18:27:25Z I! Loaded outputs: file http mqtt
2025-02-20T18:27:25Z I! Tags enabled: host=654f1341740f
2025-02-20T18:27:25Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"654f1341740f", Flush Interval:10s
2025-02-20T18:27:25Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-02-20T18:27:25Z D! [agent] Initializing plugins
2025-02-20T18:27:25Z D! [agent] Connecting outputs
2025-02-20T18:27:25Z D! [agent] Attempting connection to [outputs.http]
2025-02-20T18:27:25Z D! [agent] Successfully connected to outputs.http
2025-02-20T18:27:25Z D! [agent] Attempting connection to [outputs.mqtt]
2025-02-20T18:27:25Z D! [agent] Successfully connected to outputs.mqtt
2025-02-20T18:27:25Z D! [agent] Attempting connection to [outputs.file]
2025-02-20T18:27:25Z D! [agent] Successfully connected to outputs.file
2025-02-20T18:27:25Z D! [agent] Starting service inputs
2025-02-20T18:27:25Z I! [inputs.mqtt_consumer] Connected [mqtt://mosquitto:1883]
2025-02-20T18:27:25Z I! [inputs.mqtt_consumer] Connected [mqtt://mosquitto:1883]
2025-02-20T18:27:30Z D! [outputs.http] Buffer fullness: 2 / 10000 metrics
2025-02-20T18:27:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:30Z D! [outputs.http] Buffer fullness: 2 / 10000 metrics
2025-02-20T18:27:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 685.2µs
2025-02-20T18:27:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:35Z D! [outputs.http] Buffer fullness: 4 / 10000 metrics
2025-02-20T18:27:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:35Z D! [outputs.http] Buffer fullness: 4 / 10000 metrics
2025-02-20T18:27:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:39Z D! [outputs.http] Buffer fullness: 6 / 10000 metrics
2025-02-20T18:27:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:39Z D! [outputs.http] Buffer fullness: 6 / 10000 metrics
2025-02-20T18:27:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:45Z D! [outputs.file] Wrote batch of 2 metrics in 153.9µs
2025-02-20T18:27:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 563.5µs
2025-02-20T18:27:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:45Z D! [outputs.http] Buffer fullness: 8 / 10000 metrics
2025-02-20T18:27:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:45Z D! [outputs.http] Buffer fullness: 8 / 10000 metrics
2025-02-20T18:27:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:50Z D! [outputs.http] Buffer fullness: 10 / 10000 metrics
2025-02-20T18:27:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:50Z D! [outputs.http] Buffer fullness: 10 / 10000 metrics
2025-02-20T18:27:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:55Z D! [outputs.file] Wrote batch of 2 metrics in 144.2µs
2025-02-20T18:27:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 592.3µs
2025-02-20T18:27:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:55Z D! [outputs.http] Buffer fullness: 12 / 10000 metrics
2025-02-20T18:27:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:55Z D! [outputs.http] Buffer fullness: 12 / 10000 metrics
2025-02-20T18:27:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:00Z D! [outputs.http] Buffer fullness: 14 / 10000 metrics
2025-02-20T18:28:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:00Z D! [outputs.http] Buffer fullness: 14 / 10000 metrics
2025-02-20T18:28:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:05Z D! [outputs.file] Wrote batch of 2 metrics in 192.6µs
2025-02-20T18:28:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 511.4µs
2025-02-20T18:28:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:05Z D! [outputs.http] Buffer fullness: 16 / 10000 metrics
2025-02-20T18:28:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:05Z D! [outputs.http] Buffer fullness: 16 / 10000 metrics
2025-02-20T18:28:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:09Z D! [outputs.http] Buffer fullness: 18 / 10000 metrics
2025-02-20T18:28:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:09Z D! [outputs.http] Buffer fullness: 18 / 10000 metrics
2025-02-20T18:28:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:15Z D! [outputs.file] Wrote batch of 2 metrics in 192µs
2025-02-20T18:28:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:15Z D! [outputs.mqtt] Wrote batch of 2 metrics in 970.8µs
2025-02-20T18:28:15Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:15Z D! [outputs.http] Buffer fullness: 20 / 10000 metrics
2025-02-20T18:28:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:15Z D! [outputs.http] Buffer fullness: 20 / 10000 metrics
2025-02-20T18:28:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:20Z D! [outputs.http] Buffer fullness: 22 / 10000 metrics
2025-02-20T18:28:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:20Z D! [outputs.http] Buffer fullness: 22 / 10000 metrics
2025-02-20T18:28:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:25Z D! [outputs.file] Wrote batch of 2 metrics in 115.7µs
2025-02-20T18:28:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:25Z D! [outputs.mqtt] Wrote batch of 2 metrics in 406.9µs
2025-02-20T18:28:25Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:25Z D! [outputs.http] Buffer fullness: 24 / 10000 metrics
2025-02-20T18:28:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:25Z D! [outputs.http] Buffer fullness: 24 / 10000 metrics
2025-02-20T18:28:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:30Z D! [outputs.http] Buffer fullness: 26 / 10000 metrics
2025-02-20T18:28:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:30Z D! [outputs.http] Buffer fullness: 26 / 10000 metrics
2025-02-20T18:28:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:35Z D! [outputs.file] Wrote batch of 2 metrics in 125µs
2025-02-20T18:28:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 551µs
2025-02-20T18:28:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:35Z D! [outputs.http] Buffer fullness: 28 / 10000 metrics
2025-02-20T18:28:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:35Z D! [outputs.http] Buffer fullness: 28 / 10000 metrics
2025-02-20T18:28:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:39Z D! [outputs.http] Buffer fullness: 30 / 10000 metrics
2025-02-20T18:28:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:39Z D! [outputs.http] Buffer fullness: 30 / 10000 metrics
2025-02-20T18:28:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:45Z D! [outputs.file] Wrote batch of 2 metrics in 220.9µs
2025-02-20T18:28:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 643.6µs
2025-02-20T18:28:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:45Z D! [outputs.http] Buffer fullness: 32 / 10000 metrics
2025-02-20T18:28:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:45Z D! [outputs.http] Buffer fullness: 32 / 10000 metrics
2025-02-20T18:28:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:50Z D! [outputs.http] Buffer fullness: 34 / 10000 metrics
2025-02-20T18:28:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:50Z D! [outputs.http] Buffer fullness: 34 / 10000 metrics
2025-02-20T18:28:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:55Z D! [outputs.file] Wrote batch of 2 metrics in 199.9µs
2025-02-20T18:28:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 977.5µs
2025-02-20T18:28:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:55Z D! [outputs.http] Buffer fullness: 36 / 10000 metrics
2025-02-20T18:28:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:55Z D! [outputs.http] Buffer fullness: 36 / 10000 metrics
{"fields":{"timestamp":1740076050},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076055}
{"fields":{"timestamp":1740076050},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076055}
{"fields":{"timestamp":1740076060},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076065}
{"fields":{"timestamp":1740076060},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076065}
{"fields":{"timestamp":1740076070},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076075}
{"fields":{"timestamp":1740076070},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076075}
{"fields":{"timestamp":1740076080},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076085}
{"fields":{"timestamp":1740076080},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076085}
{"fields":{"timestamp":1740076090},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076095}
{"fields":{"timestamp":1740076090},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076095}
{"fields":{"timestamp":1740076100},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076105}
{"fields":{"timestamp":1740076100},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076105}
{"fields":{"timestamp":1740076110},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076115}
{"fields":{"timestamp":1740076110},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076115}
{"fields":{"timestamp":1740076120},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076125}
{"fields":{"timestamp":1740076120},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076125}
{"fields":{"timestamp":1740076130},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076135}
{"fields":{"timestamp":1740076130},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076135}
{"fields":{"timestamp":1740076140},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076145}
{"fields":{"timestamp":1740076140},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076145}
{"fields":{"timestamp":1740076150},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076155}
{"fields":{"timestamp":1740076160},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076165}
{"fields":{"timestamp":1740076170},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076175}
2025-02-20T18:28:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:00Z D! [outputs.http] Buffer fullness: 38 / 10000 metrics
2025-02-20T18:29:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:00Z D! [outputs.http] Buffer fullness: 38 / 10000 metrics
2025-02-20T18:29:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:05Z D! [outputs.file] Wrote batch of 2 metrics in 153.7µs
2025-02-20T18:29:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 655.3µs
2025-02-20T18:29:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:05Z D! [outputs.http] Buffer fullness: 40 / 10000 metrics
2025-02-20T18:29:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:05Z D! [outputs.http] Buffer fullness: 40 / 10000 metrics
2025-02-20T18:29:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:09Z D! [outputs.http] Buffer fullness: 42 / 10000 metrics
2025-02-20T18:29:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:09Z D! [outputs.http] Buffer fullness: 42 / 10000 metrics
2025-02-20T18:29:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:15Z D! [outputs.file] Wrote batch of 2 metrics in 212.7µs
2025-02-20T18:29:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:15Z D! [outputs.mqtt] Wrote batch of 2 metrics in 812.3µs
2025-02-20T18:29:15Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:15Z D! [outputs.http] Buffer fullness: 43 / 10000 metrics
2025-02-20T18:29:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:15Z D! [outputs.http] Buffer fullness: 43 / 10000 metrics
2025-02-20T18:29:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:20Z D! [outputs.http] Buffer fullness: 45 / 10000 metrics
2025-02-20T18:29:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:20Z D! [outputs.http] Buffer fullness: 45 / 10000 metrics
2025-02-20T18:29:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:25Z D! [outputs.file] Wrote batch of 1 metrics in 100µs
2025-02-20T18:29:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:25Z D! [outputs.mqtt] Wrote batch of 2 metrics in 518.5µs
2025-02-20T18:29:25Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:25Z D! [outputs.http] Buffer fullness: 46 / 10000 metrics
2025-02-20T18:29:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:25Z D! [outputs.http] Buffer fullness: 46 / 10000 metrics
2025-02-20T18:29:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:30Z D! [outputs.http] Buffer fullness: 48 / 10000 metrics
2025-02-20T18:29:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:30Z D! [outputs.http] Buffer fullness: 48 / 10000 metrics
2025-02-20T18:29:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:35Z D! [outputs.file] Wrote batch of 1 metrics in 109µs
2025-02-20T18:29:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 565.1µs
2025-02-20T18:29:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:35Z D! [outputs.http] Buffer fullness: 49 / 10000 metrics
2025-02-20T18:29:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:35Z D! [outputs.http] Buffer fullness: 49 / 10000 metrics
2025-02-20T18:29:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:39Z D! [outputs.http] Buffer fullness: 51 / 10000 metrics
2025-02-20T18:29:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:39Z D! [outputs.http] Buffer fullness: 51 / 10000 metrics
2025-02-20T18:29:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:45Z D! [outputs.file] Wrote batch of 1 metrics in 99.8µs
2025-02-20T18:29:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 453.7µs
2025-02-20T18:29:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:45Z D! [outputs.http] Buffer fullness: 52 / 10000 metrics
2025-02-20T18:29:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:45Z D! [outputs.http] Buffer fullness: 52 / 10000 metrics
2025-02-20T18:29:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:50Z D! [outputs.http] Buffer fullness: 54 / 10000 metrics
2025-02-20T18:29:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:50Z D! [outputs.http] Buffer fullness: 54 / 10000 metrics
2025-02-20T18:29:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:55Z D! [outputs.file] Wrote batch of 1 metrics in 124.8µs
2025-02-20T18:29:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 885.5µs
2025-02-20T18:29:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:55Z D! [outputs.http] Buffer fullness: 55 / 10000 metrics
2025-02-20T18:29:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:55Z D! [outputs.http] Buffer fullness: 55 / 10000 metrics
2025-02-20T18:29:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:00Z D! [outputs.http] Buffer fullness: 57 / 10000 metrics
2025-02-20T18:30:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:00Z D! [outputs.http] Buffer fullness: 57 / 10000 metrics
{"fields":{"timestamp":1740076180},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076185}
{"fields":{"timestamp":1740076190},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076195}
{"fields":{"timestamp":1740076200},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076205}
{"fields":{"timestamp":1740076210},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076215}
2025-02-20T18:30:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:05Z D! [outputs.file] Wrote batch of 1 metrics in 123.3µs
2025-02-20T18:30:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 716.5µs
2025-02-20T18:30:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:05Z D! [outputs.http] Buffer fullness: 58 / 10000 metrics
2025-02-20T18:30:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:05Z D! [outputs.http] Buffer fullness: 58 / 10000 metrics
2025-02-20T18:30:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:09Z D! [outputs.http] Buffer fullness: 60 / 10000 metrics
2025-02-20T18:30:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:09Z D! [outputs.http] Buffer fullness: 60 / 10000 metrics
2025-02-20T18:30:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:15Z D! [outputs.file] Wrote batch of 1 metrics in 90.8µs
2025-02-20T18:30:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:15Z D! [outputs.mqtt] Wrote batch of 2 metrics in 644.3µs
2025-02-20T18:30:15Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:15Z D! [outputs.http] Buffer fullness: 61 / 10000 metrics
2025-02-20T18:30:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:15Z D! [outputs.http] Buffer fullness: 61 / 10000 metrics
2025-02-20T18:30:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:20Z D! [outputs.http] Buffer fullness: 63 / 10000 metrics
2025-02-20T18:30:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:20Z D! [outputs.http] Buffer fullness: 63 / 10000 metrics
2025-02-20T18:30:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:25Z D! [outputs.file] Wrote batch of 1 metrics in 117.3µs
2025-02-20T18:30:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:25Z D! [outputs.mqtt] Wrote batch of 2 metrics in 905.4µs
2025-02-20T18:30:25Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:25Z D! [outputs.http] Buffer fullness: 64 / 10000 metrics
2025-02-20T18:30:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:25Z D! [outputs.http] Buffer fullness: 64 / 10000 metrics
2025-02-20T18:30:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:30Z D! [outputs.http] Buffer fullness: 66 / 10000 metrics
2025-02-20T18:30:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:30Z D! [outputs.http] Buffer fullness: 66 / 10000 metrics
2025-02-20T18:30:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:35Z D! [outputs.file] Wrote batch of 1 metrics in 81.8µs
2025-02-20T18:30:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 594.8µs
2025-02-20T18:30:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:35Z D! [outputs.http] Buffer fullness: 67 / 10000 metrics
2025-02-20T18:30:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:35Z D! [outputs.http] Buffer fullness: 67 / 10000 metrics
2025-02-20T18:30:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:39Z D! [outputs.http] Buffer fullness: 69 / 10000 metrics
2025-02-20T18:30:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:39Z D! [outputs.http] Buffer fullness: 69 / 10000 metrics
2025-02-20T18:30:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:45Z D! [outputs.file] Wrote batch of 1 metrics in 102.3µs
2025-02-20T18:30:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 876.4µs
2025-02-20T18:30:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:45Z D! [outputs.http] Buffer fullness: 70 / 10000 metrics
2025-02-20T18:30:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:45Z D! [outputs.http] Buffer fullness: 70 / 10000 metrics
2025-02-20T18:30:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:50Z D! [inputs.mqtt_consumer] Connecting [mqtt://mosquitto:1883]
2025-02-20T18:30:50Z I! [inputs.mqtt_consumer] Connected [mqtt://mosquitto:1883]
2025-02-20T18:30:50Z D! [outputs.http] Buffer fullness: 72 / 10000 metrics
2025-02-20T18:30:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:50Z D! [outputs.http] Buffer fullness: 72 / 10000 metrics
2025-02-20T18:30:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:55Z D! [outputs.file] Wrote batch of 1 metrics in 233.5µs
2025-02-20T18:30:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 756.1µs
2025-02-20T18:30:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:55Z D! [outputs.http] Buffer fullness: 73 / 10000 metrics
2025-02-20T18:30:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:55Z D! [outputs.http] Buffer fullness: 73 / 10000 metrics
2025-02-20T18:30:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:00Z D! [outputs.http] Buffer fullness: 75 / 10000 metrics
{"fields":{"timestamp":1740076220},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076225}
{"fields":{"timestamp":1740076230},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076235}
{"fields":{"timestamp":1740076240},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076245}
{"fields":{"timestamp":1740076250},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076255}
{"fields":{"timestamp":1740076260},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076265}
2025-02-20T18:31:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:00Z D! [outputs.http] Buffer fullness: 75 / 10000 metrics
2025-02-20T18:31:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:05Z D! [outputs.file] Wrote batch of 1 metrics in 127.9µs
2025-02-20T18:31:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 692µs
2025-02-20T18:31:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:05Z D! [outputs.http] Buffer fullness: 76 / 10000 metrics
2025-02-20T18:31:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:05Z D! [outputs.http] Buffer fullness: 76 / 10000 metrics
2025-02-20T18:31:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:05Z D! [agent] Stopping service inputs
2025-02-20T18:31:05Z D! [inputs.mqtt_consumer] Disconnecting [mqtt://mosquitto:1883]
2025-02-20T18:31:05Z D! [inputs.mqtt_consumer] Disconnected [mqtt://mosquitto:1883]
2025-02-20T18:31:05Z D! [inputs.mqtt_consumer] Disconnecting [mqtt://mosquitto:1883]
2025-02-20T18:31:06Z D! [inputs.mqtt_consumer] Disconnected [mqtt://mosquitto:1883]
2025-02-20T18:31:06Z D! [agent] Input channel closed
2025-02-20T18:31:06Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-02-20T18:31:06Z D! [outputs.file] Wrote batch of 1 metrics in 76.3µs
2025-02-20T18:31:06Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:06Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:06Z D! [outputs.http] Buffer fullness: 76 / 10000 metrics
2025-02-20T18:31:06Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:06Z I! [agent] Stopping running outputs
2025-02-20T18:31:06Z D! [agent] Stopped Successfully
```

### System info

Official Telegraf Docker container for 1.32 and 1.33, Mosquitto docker container 2.0, Docker 20.10.12

### Docker

```
version: '3.6'

services:

telegraf:
container_name: telegraf
image: telegraf:1.33
volumes:
- ./telegraf.conf:/etc/telegraf/telegraf.conf
restart: always
depends_on:
- mosquitto
command: ["telegraf", "--config", "/etc/telegraf/telegraf.conf", "--debug"]

mosquitto:
container_name: mosquitto
image: eclipse-mosquitto:2.0
volumes:
- ./mosquitto.conf:/mosquitto/config/mosquitto.conf
ports:
- 1883:1883

```

### Steps to reproduce

1. Have two MQTT consumers (A & B) listening on different topics. MQTT consumer B has a `max_undelivered_messages` set to a low number (in my example, it's set to 10)
2. Have both MQTT consumers sending to TWO outputs, **one of which is unresponsive / will fail to write**, and one which we'll call "good"
3. Send constant info over MQTT to both MQTT topics for A & B
4. Monitor "good" output to make sure data from A & B are flowing
5. Wait for MQTT B consumer to disconnect + reconnect
6. Monitor "good" output for data from A & B

### Expected behavior

Data from A & B continues to send to both outputs. After B disconnects, I expect it to reconnect and resume sending MQTT B messages to the "good" output.

### Actual behavior

After B disconnects, it seems to reconnect but does not send any more data to outputs.

### Additional info

This is not a resource issue, as I've confirmed that my docker containers are not even close to running out of memory / CPU. Attached is a diagram showing how I set up my telegraf.conf to reproduce the issue, but we were seeing this issue with independent MQTT messages coming in production.

![Image](https://github.com/user-attachments/assets/5461086f-df09-4fc8-b226-e2bafcd6f0a2)

Here is a link to my Github repo with a docker compose, mosquitto.conf, and telegraf.conf that repro the issue reliably. https://github.com/annamooseity/telegraf-mqtt-weirdness

Contributor guide

Open the contributing guide

Research direction

Reproduce the supplied configuration with the mqtt_consumer inputs and http, mqtt, and file outputs, using the logs to observe buffering and delivery. Trace the agent path handling a failing outputs.http write alongside successful outputs.file and outputs.mqtt writes; done means one unresponsive output no longer prevents the MQTT inputs from continuing to report.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.