log gets very noisy in cases the OTelExporter can not write successfully
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 616
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 10
Description
Describe the bug
In setups with a running and reachable OpenTelemetry Backend, that does for some reason not work correctly, the log gets flooded with messages about this.
To Reproduce
Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.
- Run icinga2 and an otlp backend
- configure them to use opentelemetry
- have the opentelemetry backend reachable, but not accept the data
- look at the icinga2.log, find entries like for example this
[2026-07-22 13:13:34 +0000] critical/OTelExporter: Error: Connection reset by peer [system:104 at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:134 in function 'do_complete']
[2026-07-22 13:13:34 +0000] information/OTelExporter: Connecting to OpenTelemetry backend on host '127.0.0.1:9090'.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Successfully connected to OpenTelemetry backend.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Connecting to OpenTelemetry backend on host '127.0.0.1:9090'.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Successfully connected to OpenTelemetry backend.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Connecting to OpenTelemetry backend on host '127.0.0.1:9090'.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Successfully connected to OpenTelemetry backend.
[2026-07-22 13:13:34 +0000] warning/OTelExporter: Unexpected Content-Type from OpenTelemetry backend 'text/plain; charset=utf-8' (Bad Request):
Icinga2 is a bit noisy in such cases: cat /var/log/icinga2/icinga2.log | grep "13:13:34" | grep "OTelExporter" | wc -l 1549
Expected behavior
error messages, but not too many.
Your Environment
Include as many relevant details about the environment you experienced the problem in
- Version used (
icinga2 --version): 2.16.4-1 - Operating System and version: Debian 13
- Enabled features (
icinga2 feature list): api, checker, icingadb, mainlog, notification, otlpmetrics - Icinga Web 2 version and modules (System - About): 2.14.0
- Config validation (
icinga2 daemon -C):
icinga2 daemon -C
[2026-07-22 13:20:03 +0000] information/cli: Icinga application loader (version: r2.16.4-1)
[2026-07-22 13:20:03 +0000] information/cli: Loading configuration file(s).
[2026-07-22 13:20:03 +0000] information/ConfigItem: Committing config item(s).
[2026-07-22 13:20:03 +0000] information/ApiListener: My API identity: monitor2
[2026-07-22 13:20:03 +0000] warning/ApiListener: Security notice:
Currently, all ApiUsers are allowed to use Icinga 2 DSL filter expressions
in API queries because enforce_filter_expression_permission is set to false.
This can pose a security risk as filters are evaluated within the Icinga 2
process and their complexity can be used for denial of service attacks. The
new 'filter-expression' permission can be used to allow this for individual
ApiUsers, which should only be granted to trusted users. It is recommended
to set enforce_filter_expression_permission to true to enforce the
permission. This will become the default in v2.17.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 11 Hosts.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 10 Zones.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 9 Endpoints.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 OTLPMetricsWriter.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 264 CheckCommands.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 32 Services.
[2026-07-22 13:20:03 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2026-07-22 13:20:03 +0000] information/cli: Finished validating the configuration file(s).
- If you run multiple Icinga 2 instances, the
zones.conffile (oricinga2 object list --type Endpointandicinga2 object list --type Zone) from all affected nodes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with icinga2 2.16.4-1, the otlpmetrics feature, and a reachable OpenTelemetry backend that rejects data. Inspect the OTelExporter log sequence and the reported 1,549 entries in one second. Done means failed exports still produce useful error information without flooding icinga2.log.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 48/100