influxdata / influxdata/telegraf
Kafka output occasionally sends messages with an extra trailing newline (LF), causing CSV parsing failures in downstream consumers
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "5s"
precision = "0s"
debug = true
[[outputs.kafka]]
brokers = ["redpanda-0:9092"]
topic = "telegraf"
routing_tag = "host"
sasl_username = "superuser"
sasl_password = "secretpassword"
sasl_mechanism = "SCRAM-SHA-256"
data_format = "csv"
[[inputs.exec]]
commands = ["/opt/some_metric.py"]
timeout = "15s"
data_format = "csv"
name_override = "some_metric"
csv_column_names = ["column1", "column2", "column3", "column4", "column5"]
```
### Logs from Telegraf
```text
2025-10-11T07:59:28Z I! Loading config: /etc/telegraf/telegraf.conf
2025-10-11T07:59:28Z I! Starting Telegraf 1.36.2 brought to you by InfluxData the makers of InfluxDB
2025-10-11T07:59:28Z I! Available plugins: 239 inputs, 9 aggregators, 35 processors, 26 parsers, 65 outputs, 6 secret-stores
2025-10-11T07:59:28Z I! Loaded inputs: exec
2025-10-11T07:59:28Z I! Loaded aggregators:
2025-10-11T07:59:28Z I! Loaded processors:
2025-10-11T07:59:28Z I! Loaded secretstores:
2025-10-11T07:59:28Z I! Loaded outputs: kafka
2025-10-11T07:59:28Z I! Tags enabled: host=a9aab275aa6f
2025-10-11T07:59:28Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"a9aab275aa6f", Flush Interval:10s
2025-10-11T07:59:28Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-10-11T07:59:28Z D! [agent] Initializing plugins
2025-10-11T07:59:28Z D! [agent] Connecting outputs
2025-10-11T07:59:28Z D! [agent] Attempting connection to [outputs.kafka]
2025-10-11T07:59:28Z D! [agent] Successfully connected to outputs.kafka
2025-10-11T07:59:28Z D! [agent] Starting service inputs
2025-10-11T07:59:38Z D! [outputs.kafka] Wrote batch of 1 metrics in 3.939189ms
2025-10-11T07:59:38Z D! [outputs.kafka] Buffer fullness: 0 / 10000 metrics
2025-10-11T07:59:49Z D! [outputs.kafka] Wrote batch of 1 metrics in 549.29µs
2025-10-11T07:59:49Z D! [outputs.kafka] Buffer fullness: 0 / 10000 metrics
2025-10-11T08:00:04Z D! [outputs.kafka] Wrote batch of 2 metrics in 651.148µs
```
### System info
Telegraf 1.36.2, Podman/Docker docker.io/library/telegraf:1.36.2
### Docker
I'm using Telegraf with the exec input (CSV format) and kafka output (also CSV format). My script outputs exactly one well-formed CSV line followed by a single newline (\n), as confirmed by od -c and wc -l.
However, approximately every 5 minutes, Telegraf appears to send a Kafka message that contains two newlines — i.e., the valid CSV line followed by an empty line (\n\n). This causes downstream consumers (e.g., ClickHouse Kafka engine) to fail with parsing errors like:
```
ERROR: text "" is not like UInt32
```
Code of some_metric.py:
```
#!/usr/bin/env python3
import random
duration = random.uniform(0.000000000000000001, 9.999999999999999999)
print("foo.bar",duration,"OK","0","Success",sep=",")
```
### Steps to reproduce
1. Run telegraf with with included config
2. Check messages in kafka-ui / redpanda-console
### Expected behavior
Every Kafka message should contain exactly one CSV record terminated by a single \n, matching the script’s output.
### Actual behavior
The script consistently outputs only one line (verified via script.sh | wc -l → always 1).
The issue occurs periodically (~every 5 minutes), some messages ends with \n\n instead of \n, and so clickhouse Kafka engine can't parse messages
### Additional info
List of filtered messages with \n\n:
Filtered messages content:
Contributor guide
Research direction
Start by tracing the exec input and Kafka output paths for CSV handling, using the supplied configuration and the Python script as the reproduction. Verify where line endings are added or preserved, then confirm that each Kafka message contains one CSV record ending in exactly one newline without breaking normal batches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka, python
- Domain
- data, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100