Downtime/Comment legacy_id is unsafe

Describe the bug

During some investigation why Downtimes can not be deleted from Icinga Web 2 I noticed that the legacy_id is not very safely implemented.

Meaning:

• It will change on every reload, but is synced to the cluster partner
• Will only be set correctly in the IDO on a reload
• Internal next counter and map will not be in sync

When it is not a problem:

• Icinga Web 2 sends commands via API
• Icinga 2 master zone is single instance

Still a problem:

• IDs can be duplicated in IDO and state of Icinga 2 (until reload)

To Reproduce

Untested but likely the way to reproduce:

1. Reload node-1 without reloading the node-2 which usually writes the IDO
2. l_LegacyDowntimesCache will differ on node-1
3. Send DEL_SVC_DOWNTIME;XXX and it should delete an unrelated downtime

Expected behavior

While command pipe and the legacy IDs are in use by IDO and Icinga Web 2 accessing IDO is in use this is a pending problem.

Legacy ID handling is only obsolete when fully switching to Icinga DB and API.

Logs

information/ExternalCommandListener: Executing external command: [1644316000] SCHEDULE_SVC_DOWNTIME;icinga-1;test-service;1644315995;1644319595;1;0;0;icingaadmin;Test
information/Downtime: Triggering downtime 'icinga-1!test-service!4ea6fb60-cd48-4d38-86ee-c817ce5013ff' for checkable 'icinga-1!test-service'.
information/ConfigObjectUtility: Created and activated object 'icinga-1!test-service!4ea6fb60-cd48-4d38-86ee-c817ce5013ff' of type 'Downtime'.
information/Downtime: Added downtime 'icinga-1!test-service!4ea6fb60-cd48-4d38-86ee-c817ce5013ff' between '2022-02-08 11:26:35' and '2022-02-08 12:26:35', author: 'icingaadmin', fixed
...
information/ExternalCommandListener: Executing external command: [1644316022] DEL_SVC_DOWNTIME;895
information/ConfigObjectUtility: Deleted object 'unrelated-host!other-service!97646e49-f27f-48b3-af41-17b5b365bd29' of type 'Downtime'.
information/Downtime: Removed downtime 'unrelated-host!other-service!97646e49-f27f-48b3-af41-17b5b365bd29' from checkable 'unrelated-host!other-service' (Reason: cancelled by user).

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): 2.12.5
• Operating System and version: RHEL 7
• Icinga Web 2 version and modules (System - About): 2.8.5

ref/NC/739102

Start by reproducing the documented node-1 reload/node-2 and DEL_SVC_DOWNTIME sequence, then trace legacy_id handling around l_LegacyDowntimesCache, the external command, and IDO synchronization. Done means a legacy ID cannot delete an unrelated downtime and remains consistent across reloads, cluster nodes, and IDO access.