influxdata / influxdata/telegraf

Syslog plugin will send multiple messages to Opentsdb and Riemann outputs

Open
#17,288 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
# Global tags can be specified here in key="value" format.
[global_tags]
# dc = "us-east-1" # will tag all metrics with dc=us-east-1
# rack = "1a"
## Environment variables can be used as tags, and throughout the config file
# user = "$USER"

# Configuration for telegraf agent
[agent]
## Default data collection interval for all inputs
interval = "10s"
## Rounds collection interval to 'interval'
## ie, if interval="10s" then always collect on :00, :10, :20, etc.
round_interval = true

## Telegraf will send metrics to outputs in batches of at most
## metric_batch_size metrics.
## This controls the size of writes that Telegraf sends to output plugins.
metric_batch_size = 1000

## Maximum number of unwritten metrics per output. Increasing this value
## allows for longer periods of output downtime without dropping metrics at the
## cost of higher maximum memory usage.
metric_buffer_limit = 10000

## Collection jitter is used to jitter the collection by a random amount.
## Each plugin will sleep for a random time within jitter before collecting.
## This can be used to avoid many plugins querying things like sysfs at the
## same time, which can have a measurable effect on the system.
collection_jitter = "0s"

## Default flushing interval for all outputs. Maximum flush_interval will be
## flush_interval + flush_jitter
flush_interval = "10s"
## Jitter the flush interval by a random amount. This is primarily to avoid
## large write spikes for users running a large number of telegraf instances.
## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
flush_jitter = "0s"

## By default or when set to "0s", precision will be set to the same
## timestamp order as the collection interval, with the maximum being 1s.
## ie, when interval = "10s", precision will be "1s"
## when interval = "250ms", precision will be "1ms"
## Precision will NOT be used for service inputs. It is up to each individual
## service input to set the timestamp at the appropriate precision.
## Valid time units are "ns", "us" (or "µs"), "ms", "s".
precision = ""

## Log at debug level.
# debug = false
## Log only error level messages.
# quiet = false

## Log target controls the destination for logs and can be one of "file",
## "stderr" or, on Windows, "eventlog". When set to "file", the output file
## is determined by the "logfile" setting.
# logtarget = "file"

## Name of the file to be logged to when using the "file" logtarget. If set to
## the empty string then logs are written to stderr.
# logfile = ""

## The logfile will be rotated after the time interval specified. When set
## to 0 no time based rotation is performed. Logs are rotated only when
## written to, if there is no log activity rotation may be delayed.
# logfile_rotation_interval = "0d"

## The logfile will be rotated when it becomes larger than the specified
## size. When set to 0 no size based rotation is performed.
# logfile_rotation_max_size = "0MB"

## Maximum number of rotated archives to keep, any older logs are deleted.
## If set to -1, no archives are removed.
# logfile_rotation_max_archives = 5

## Override default hostname, if empty use os.Hostname()
hostname = ""
## If set to true, do no set the "host" tag in the telegraf agent.
omit_hostname = false

###############################################################################
# OUTPUT PLUGINS #
###############################################################################

[[outputs.file]]
files = ["stdout"]

#[[outputs.influxdb]]
# urls = ["http://127.0.0.1:8086"]
# database = "monitoring_syslog"

#[[outputs.riemann]]
# ## The full TCP or UDP URL of the Riemann server
# url = "tcp://127.0.0.1:5555"
#
## ## Riemann event TTL, floating-point time in seconds.
## ## Defines how long that an event is considered valid for in Riemann
# ttl = 10.0
##
## ## Separator to use between measurement and field name in Riemann service name
## ## This does not have any effect if 'measurement_as_attribute' is set to 'true'
# separator = "/"
##
## ## Set measurement name as Riemann attribute 'measurement', instead of prepending it to the Riemann service name
# measurement_as_attribute = false
##
## ## Send string metrics as Riemann event states.
## ## Unless enabled all string metrics will be ignored
# string_as_state = true
##
## ## A list of tag keys whose values get sent as Riemann tags.
## ## If empty, all Telegraf tag values will be sent as tags
## # tag_keys = ["telegraf","custom_tag"]
##
# ## Additional Riemann tags to send.
# tags = ["telegraf"]
##
## ## Description for Riemann event
## # description_text = "metrics collected from telegraf"
##
## ## Riemann client write timeout, defaults to "5s" if not set.
## # timeout = "5s"

###############################################################################
# INPUT PLUGINS #
###############################################################################

[[inputs.syslog]]
## Protocol, address and port to host the syslog receiver.
## If no host is specified, then localhost is used.
## If no port is specified, 6514 is used (RFC5425#section-4.1).
## ex: server = "tcp://localhost:6514"
## server = "udp://:6514"
## server = "unix:///var/run/telegraf-syslog.sock"
## When using tcp, consider using 'tcp4' or 'tcp6' to force the usage of IPv4
## or IPV6 respectively. There are cases, where when not specified, a system
## may force an IPv4 mapped IPv6 address.
server = "tcp://127.0.0.1:6514"

## Permission for unix sockets (only available on unix sockets)
## This setting may not be respected by some platforms. To safely restrict
## permissions it is recommended to place the socket into a previously
## created directory with the desired permissions.
## ex: socket_mode = "777"
# socket_mode = ""

## Maximum number of concurrent connections (only available on stream sockets like TCP)
## Zero means unlimited.
# max_connections = 0

## Read timeout (only available on stream sockets like TCP)
## Zero means unlimited.
# read_timeout = "0s"

## Optional TLS configuration (only available on stream sockets like TCP)
#tls_cert = "/etc/telegraf/tls.crt"
#tls_key = "/etc/telegraf/tls.key"
## Enables client authentication if set.
# tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]

## Maximum socket buffer size (in bytes when no unit specified)
## For stream sockets, once the buffer fills up, the sender will start
## backing up. For datagram sockets, once the buffer fills up, metrics will
## start dropping. Defaults to the OS default.
# read_buffer_size = "64KiB"

## Period between keep alive probes (only applies to TCP sockets)
## Zero disables keep alive probes. Defaults to the OS configuration.
# keep_alive_period = "5m"

## Content encoding for message payloads
## Can be set to "gzip" for compressed payloads or "identity" for no encoding.
# content_encoding = "identity"

## Maximum size of decoded packet (in bytes when no unit specified)
# max_decompression_size = "500MB"

## Framing technique used for messages transport
## Available settings are:
## octet-counting -- see RFC5425#section-4.3.1 and RFC6587#section-3.4.1
## non-transparent -- see RFC6587#section-3.4.2
# framing = "octet-counting"

## The trailer to be expected in case of non-transparent framing (default = "LF").
## Must be one of "LF", or "NUL".
# trailer = "LF"

## Whether to parse in best effort mode or not (default = false).
## By default best effort parsing is off.
# best_effort = false

## The RFC standard to use for message parsing
## By default RFC5424 is used. RFC3164 only supports UDP transport (no streaming support)
## Must be one of "RFC5424", or "RFC3164".
syslog_standard = "RFC5424"

## Character to prepend to SD-PARAMs (default = "_").
## A syslog message can contain multiple parameters and multiple identifiers within structured data section.
## Eg., [id1 name1="val1" name2="val2"][id2 name1="val1" nameA="valA"]
## For each combination a field is created.
## Its name is created concatenating identifier, sdparam_separator, and parameter name.
# sdparam_separator = "_"
```

### Logs from Telegraf

```text
(pasting riemann messages directly down below)
```

### System info

Telegraf 1.35.1, Rocky Linux release 8.10 (Green Obsidian) x86_64

### Docker

_No response_

### Steps to reproduce

I've recently tested Telegraf syslog plugin (by redirecting all messages from local rsyslog to Telegraf instance), configured to use riemann output plugin.

What I've noticed is that, when rsyslog sends a message, which is then forwarded to Telegraf, it will split that message to multiple events and sent them to Riemann. For example, this message:

```
echo "57 <13>1 2018-10-01T12:00:00.0Z example.org root - - - test" | nc 127.0.0.1 6514
```

will be sent as:

```
#riemann.codec.Event{:host monitor.verser.info, :service syslog/severity_code, :state ok, :description nil, :metric 5, :tags [telegraf root user example.org notice 127.0.0.1], :time 1751652266, :ttl 10.0, :appname root, :facility user, :hostname example.org, :severity notice, :source 127.0.0.1}

#riemann.codec.Event{:host monitor.verser.info, :service syslog/version, :state ok, :description nil, :metric 1, :tags [telegraf root user example.org notice 127.0.0.1], :time 1751652266, :ttl 10.0, :severity notice, :source 127.0.0.1, :appname root, :facility user, :hostname example.org}

#riemann.codec.Event{:host monitor.verser.info, :service syslog/timestamp, :state ok, :description nil, :metric 1538395200000000000, :tags [telegraf root user example.org notice 127.0.0.1], :time 1751652266, :ttl 10.0, :hostname example.org, :severity notice, :source 127.0.0.1, :appname root, :facility user}

#riemann.codec.Event{:host monitor.verser.info, :service syslog/message, :state test, :description nil, :metric 0.0, :tags [telegraf root user example.org notice 127.0.0.1], :time 1751652266, :ttl 10.0, :appname root, :facility user, :hostname example.org, :severity notice, :source 127.0.0.1}

#riemann.codec.Event{:host monitor.verser.info, :service syslog/facility_code, :state ok, :description nil, :metric 1, :tags [telegraf root user example.org notice 127.0.0.1], :time 1751652266, :ttl 10.0, :source 127.0.0.1, :appname root, :facility user, :hostname example.org, :severity notice}
```

I've tested with Opentsdb output, which is then passed to Riemann; pretty much the same behavior. However, when I set Telegraf to use influxdb output, it will send it as a single message.

I'm not faimilar with Telegraf internals, but why this is happening? When Riemann receive this as a multiple messages, it will need to reconstruct it back and that isn't always the best approach - in case of high load, different messages can intervene, making reconstruction very hard.

### Expected behavior

Riemann output should receive single message for a single syslog line.

### Actual behavior

Riemann output receive multiple messages.

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the syslog input represents one parsed line and how the Riemann and OpenTSDB outputs serialize fields into events. Compare that path with the InfluxDB output, using the reproduction message and configuration in the issue. Done means one syslog line reaches Riemann and OpenTSDB as one message rather than separate field events.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.