Icinga / Icinga/icinga2

log gets very noisy in cases the OTelExporter can not write successfully

Open
#10,962 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
2.2k
Forks
616
Avg merge
5d 6h
Merged PRs (30d)
10

Description

Describe the bug

In setups with a running and reachable OpenTelemetry Backend, that does for some reason not work correctly, the log gets flooded with messages about this.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.

  1. Run icinga2 and an otlp backend
  2. configure them to use opentelemetry
  3. have the opentelemetry backend reachable, but not accept the data
  4. look at the icinga2.log, find entries like for example this
[2026-07-22 13:13:34 +0000] critical/OTelExporter: Error: Connection reset by peer [system:104 at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:134 in function 'do_complete']
[2026-07-22 13:13:34 +0000] information/OTelExporter: Connecting to OpenTelemetry backend on host '127.0.0.1:9090'.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Successfully connected to OpenTelemetry backend.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Connecting to OpenTelemetry backend on host '127.0.0.1:9090'.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Successfully connected to OpenTelemetry backend.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Connecting to OpenTelemetry backend on host '127.0.0.1:9090'.
[2026-07-22 13:13:34 +0000] information/OTelExporter: Successfully connected to OpenTelemetry backend.
[2026-07-22 13:13:34 +0000] warning/OTelExporter: Unexpected Content-Type from OpenTelemetry backend 'text/plain; charset=utf-8' (Bad Request):

Icinga2 is a bit noisy in such cases: cat /var/log/icinga2/icinga2.log | grep "13:13:34" | grep "OTelExporter" | wc -l 1549

Expected behavior

error messages, but not too many.

Your Environment

Include as many relevant details about the environment you experienced the problem in

  • Version used (icinga2 --version): 2.16.4-1
  • Operating System and version: Debian 13
  • Enabled features (icinga2 feature list): api, checker, icingadb, mainlog, notification, otlpmetrics
  • Icinga Web 2 version and modules (System - About): 2.14.0
  • Config validation (icinga2 daemon -C):
icinga2 daemon -C
[2026-07-22 13:20:03 +0000] information/cli: Icinga application loader (version: r2.16.4-1)
[2026-07-22 13:20:03 +0000] information/cli: Loading configuration file(s).
[2026-07-22 13:20:03 +0000] information/ConfigItem: Committing config item(s).
[2026-07-22 13:20:03 +0000] information/ApiListener: My API identity: monitor2
[2026-07-22 13:20:03 +0000] warning/ApiListener: Security notice:
    Currently, all ApiUsers are allowed to use Icinga 2 DSL filter expressions
    in API queries because enforce_filter_expression_permission is set to false.
    This can pose a security risk as filters are evaluated within the Icinga 2
    process and their complexity can be used for denial of service attacks. The
    new 'filter-expression' permission can be used to allow this for individual
    ApiUsers, which should only be granted to trusted users. It is recommended
    to set enforce_filter_expression_permission to true to enforce the
    permission. This will become the default in v2.17.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 11 Hosts.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 10 Zones.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 9 Endpoints.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 1 OTLPMetricsWriter.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 264 CheckCommands.
[2026-07-22 13:20:03 +0000] information/ConfigItem: Instantiated 32 Services.
[2026-07-22 13:20:03 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2026-07-22 13:20:03 +0000] information/cli: Finished validating the configuration file(s).
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with icinga2 2.16.4-1, the otlpmetrics feature, and a reachable OpenTelemetry backend that rejects data. Inspect the OTelExporter log sequence and the reported 1,549 entries in one second. Done means failed exports still produce useful error information without flooding icinga2.log.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.