Icinga / Icinga/icinga2

warning/JsonRpcConnection: Error while sending JSON-RPC message for identity (satellite) [2.13.7]

Open
#9,788 10 comments 0 reactions 1 assignee View on GitHub

@petr-fischer is already working on this.

Since Aug 22, 2023.

area/distributed needs feedback
Dominant language
C++
Stars
2.2k
Forks
616
Avg merge
5d 6h
Merged PRs (30d)
10

Description

Describe the bug

Our master-satellite sync is breaking with this error/stacktrace (from the master icinga2.log):

[2023-06-12 16:58:32 +0200] information/ApiListener: Reconnecting to endpoint 'xxx-satellite.domain.cz' via host 'xxx.yyy.zzz' and port '5665'
[2023-06-12 16:58:32 +0200] information/ApiListener: New client connection for identity 'xxx-satellite.domain.cz' to [xxx.yyy.zzz]:5665
[2023-06-12 16:58:32 +0200] information/ApiListener: Finished reconnecting to endpoint 'xxx-satellite.domain.cz' via host 'xxx.yyy.zzz' and port '5665'
[2023-06-12 16:58:32 +0200] information/ApiListener: Sending config updates for endpoint 'xxx-satellite.domain.cz' in zone 'xxx'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Syncing configuration files for zone 'xxx' to endpoint 'xxx-satellite.domain.cz'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'xxx-satellite.domain.cz'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Finished sending config file updates for endpoint 'xxx-satellite.domain.cz' in zone 'xxx'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Syncing runtime objects to endpoint 'xxx-satellite.domain.cz'.
[2023-06-12 16:58:32 +0200] warning/JsonRpcConnection: Error while sending JSON-RPC message for identity 'xxx-satellite.domain.cz'
Error: Connection reset by peer

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# 0x000000000086B7BA in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpcConnection::WriteOutgoingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 3# 0x0000000000AE4456 in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000AE5910 in /usr/lib64/icinga2/sbin/icinga2
 5# make_fcontext in /lib64/libboost_context.so.1.69.0
[2023-06-12 16:58:32 +0200] warning/JsonRpcConnection: API client disconnected for identity 'xxx-satellite.domain.cz'
[2023-06-12 16:58:32 +0200] warning/ApiListener: Removing API client for endpoint 'xxx-satellite.domain.cz'. 0 API clients left.
[2023-06-12 16:58:32 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'xxx-satellite.domain.cz'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'xxx-satellite.domain.cz' in zone 'xxx'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Sending replay log for endpoint 'xxx-satellite.domain.cz' in zone 'xxx'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Finished sending replay log for endpoint 'xxx-satellite.domain.cz' in zone 'xxx'.
[2023-06-12 16:58:32 +0200] information/ApiListener: Finished syncing endpoint 'xxx-satellite.domain.cz' in zone 'xxx'.

Facts:

  • error in the log occurs on the master
  • 'xxx-satellite.domain.cz' is the satellite
  • 'xxx' zone is the zone for hosts and checks, that are running on the xxx-satellite.domain.cz satellite
  • both master and satellite are on the latest icinga 2.13.7 version
  • after satellite restart (systemctl reload icinga2), error disappears from the master icinga2.log for a moment, the next several synchronisations are OK, but within about an hour the error starts to reappear in the log
  • when this error occurs in the master log, our services from zone 'xxx' are starting to switch to the UNKNOWN (purple) state in the icinga web or they seems like stopped executing
  • we have also another, not yet updated satellite still on version 2.10.7, and there is no error messages for this old satellite in the master

To Reproduce

I don't know how to reproduce it.
If you need our exact configuration of master, satellites and agents, we can anonymize it and send, but it's probably not necessary.

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.7-1)

Copyright (c) 2012-2023 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.83.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 11.2.1
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-pgsql influxdb mainlog notification
Icinga Web 2 version and modules
Icinga Web 2 Version 	2.11.4
Git commit 	11453bfa92a70a44efbf7f966f5e7f27e9300a28
PHP Version 	7.3.33
Git commit date 	2023-01-26

Loaded Libraries
icinga/icinga-php-library 	0.11.0
icinga/icinga-php-thirdparty 	0.11.0

Loaded Modules
director 		1.10.2 	Configure
fileshipper 		1.1.0 	Configure
grafana 		1.3.6 	Configure
incubator 		0.20.0 	Configure
monitoring 		2.11.4 	Configure
icinga2 object list --type Endpoint

Output is over 6000 lines, too long.

icinga2 object list --type Zone

Output is over 6000 lines, too long.

yum list installed | grep boost (boost libs)
boost-program-options.x86_64
boost-regex.x86_64          1.53.0-28.el7     @CentOS7_os_x86_64     
boost-system.x86_64         1.53.0-28.el7     @CentOS7_os_x86_64     
boost-thread.x86_64         1.53.0-28.el7     @CentOS7_os_x86_64     
boost169-chrono.x86_64      1.69.0-2.el7      @epel                             
boost169-context.x86_64     1.69.0-2.el7      @epel                             
boost169-coroutine.x86_64   1.69.0-2.el7      @epel                             
boost169-date-time.x86_64   1.69.0-2.el7      @epel                             
boost169-filesystem.x86_64  1.69.0-2.el7      @epel                             
boost169-iostreams.x86_64   1.69.0-2.el7      @DataSpring_EPEL_7Server_x86_64   
boost169-program-options.x86_64
boost169-regex.x86_64       1.69.0-2.el7      @epel                             
boost169-system.x86_64      1.69.0-2.el7      @epel                             
boost169-thread.x86_64      1.69.0-2.el7      @epel

Additional context

There are similar bugs here, like #9153, but the error is different (broken pipe).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.