Icinga / Icinga/icinga2

Service history of checks, applied directly to masters, is mixed up between each other

Open
#10,835 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
2.2k
Forks
616
Avg merge
5d 6h
Merged PRs (30d)
10

Description

Describe the bug

The service history of checks, which are applied directly to both ha-masters, is reproducable mixed up.
So far I observed this behaviour only with checks between both masters.

Image

To Reproduce

  1. have an ha-setup
  2. apply "random" to both masters
  3. the history of both checks should mix up

Expected behavior

No mixup of service histories should happen.

Screenshots

I observed this the first time during an outage when I looked at another check, which was applied directly to both masters.
In this case, the complete history of this check on master02 is stuffed into the history of the same check at master01:

Image Image

yesterday I reproduced it with "random".
here are pieces of the check on master01 mixed into the history of the check on master02:

Image

(on a sidenote: the initial state change is counted as 2 / 3)

Image Image Image

Your Environment

Include as many relevant details about the environment you experienced the problem in

  • Version used (icinga2 --version): 2.15.2
  • Operating System and version: RHEL 8.10
  • Enabled features (icinga2 feature list):
Image
  • Icinga Web 2 version and modules (System - About): 2.12.5
  • Config validation (icinga2 daemon -C):
Image

Additional context

  • icingadb 1.5.1
  • redis 8.2.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue in a two-master HA setup with the "random" check and inspect how Icinga 2 records and synchronizes service history between masters. Compare each master's history for checks applied directly to both nodes; done means histories remain isolated and the initial state change is counted correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
43/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.