Icinga / Icinga/icinga2

Endpoints may miss events during initial config sync/replay logs

Open
#10,030 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/distributed bug
Dominant language
C++
Stars
2.2k
Forks
616
Avg merge
5d 6h
Merged PRs (30d)
10

Description

Describe the bug

When the master has to replay a huge number of logs, which can take up to two to three minutes or even more, the remote peer will never receive live generated events from the local endpoint and may lose important events.

ApiListener::RelayMessageOne() determines, as stated in its doc block, whether a message has been successfully relayed or not. This allows the caller to decide whether the message should be saved to disk or not (replay log).
https://github.com/Icinga/icinga2/blob/9e31b8b5590c6d67b7dd538b2c884bd377a4e486/lib/remote/apilistener.cpp#L1211-L1214

Once it reaches this point, it is considered as being relayed yet before it calls ApiListener::SyncSendMessage(). Keep in mind, it hasn't been checked whether the target endpoint is in syncing state so far.
https://github.com/Icinga/icinga2/blob/9e31b8b5590c6d67b7dd538b2c884bd377a4e486/lib/remote/apilistener.cpp#L1306-L1308

ApiListener::SyncSendMessage() ultimately checks whether the message should be forwarded or not, i.e. when the endpoint is in syncing state, it will simply ignore this message and never try it again. Unless there are unconnected endpoints, this message will not be cached and the endpoint will never get to see this event.
https://github.com/Icinga/icinga2/blob/9e31b8b5590c6d67b7dd538b2c884bd377a4e486/lib/remote/apilistener.cpp#L1179-L1183

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/remote/apilistener.cpp with ApiListener::RelayMessageOne() and follow its call to SyncSendMessage(). Trace how syncing endpoints are handled and how replay-log persistence is decided. Done means live endpoint events are not discarded during initial configuration sync or replay, and the affected relay behavior is covered by the project's relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.