Endpoints may miss events during initial config sync/replay logs
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 616
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 10
Description
Describe the bug
When the master has to replay a huge number of logs, which can take up to two to three minutes or even more, the remote peer will never receive live generated events from the local endpoint and may lose important events.
ApiListener::RelayMessageOne() determines, as stated in its doc block, whether a message has been successfully relayed or not. This allows the caller to decide whether the message should be saved to disk or not (replay log).
https://github.com/Icinga/icinga2/blob/9e31b8b5590c6d67b7dd538b2c884bd377a4e486/lib/remote/apilistener.cpp#L1211-L1214
Once it reaches this point, it is considered as being relayed yet before it calls ApiListener::SyncSendMessage(). Keep in mind, it hasn't been checked whether the target endpoint is in syncing state so far.
https://github.com/Icinga/icinga2/blob/9e31b8b5590c6d67b7dd538b2c884bd377a4e486/lib/remote/apilistener.cpp#L1306-L1308
ApiListener::SyncSendMessage() ultimately checks whether the message should be forwarded or not, i.e. when the endpoint is in syncing state, it will simply ignore this message and never try it again. Unless there are unconnected endpoints, this message will not be cached and the endpoint will never get to see this event.
https://github.com/Icinga/icinga2/blob/9e31b8b5590c6d67b7dd538b2c884bd377a4e486/lib/remote/apilistener.cpp#L1179-L1183
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/remote/apilistener.cpp with ApiListener::RelayMessageOne() and follow its call to SyncSendMessage(). Trace how syncing endpoints are handled and how replay-log persistence is decided. Done means live endpoint events are not discarded during initial configuration sync or replay, and the affected relay behavior is covered by the project's relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100