False positive in UsageSanityChecker

The bug we're hitting is in a different subsystem: the Usage Sanity Check (UsageSanityChecker.java). Specifically the "snapshot after removed" check, whose query is roughly:

SELECT count(*)

FROM   cloud_usage.cloud_usage cu

JOIN   cloud.snapshots s ON cu.usage_id = s.id

WHERE  cu.usage_type = 9

  AND  cu.start_date > s.removed;

i.e. usage records in cloud_usage.cloud_usage with a start_date AFTER the snapshot was already soft-deleted in cloud.snapshots. This drives the Usage Sanity Check failed mailer and feeds our Prometheus exporter.

On the running 4.22.1.0 (which includes 83ce006) we still see:

check | count
-- | --
snapshot_after_removed | 160
volume_after_removed | 36
template_after_removed | 0
vm_after_destroyed | 0

_Originally posted by @PPisz in https://github.com/apache/cloudstack/discussions/13398#discussioncomment-17321658_

Comience en UsageSanityChecker.java siguiendo la comprobación «snapshot after removed» y su consulta SQL contra cloud_usage.cloud_usage y cloud.snapshots. Investigue por qué la instancia 4.22.1.0 en ejecución informa de 160 registros y, a continuación, verifique que se haya resuelto el problema del recuento de falsos positivos sin romper el mailer de Usage Sanity Check ni la salida del exportador de Prometheus.

Escrito por el modelo de indexación a partir del texto del issue.