False positive in UsageSanityChecker

The bug we're hitting is in a different subsystem: the Usage Sanity Check (UsageSanityChecker.java). Specifically the "snapshot after removed" check, whose query is roughly:

SELECT count(*)

FROM   cloud_usage.cloud_usage cu

JOIN   cloud.snapshots s ON cu.usage_id = s.id

WHERE  cu.usage_type = 9

  AND  cu.start_date > s.removed;

i.e. usage records in cloud_usage.cloud_usage with a start_date AFTER the snapshot was already soft-deleted in cloud.snapshots. This drives the Usage Sanity Check failed mailer and feeds our Prometheus exporter.

On the running 4.22.1.0 (which includes 83ce006) we still see:

check | count
-- | --
snapshot_after_removed | 160
volume_after_removed | 36
template_after_removed | 0
vm_after_destroyed | 0

_Originally posted by @PPisz in https://github.com/apache/cloudstack/discussions/13398#discussioncomment-17321658_

UsageSanityChecker.java から始めて、「snapshot after removed」チェックと、cloud_usage.cloud_usage および cloud.snapshots に対するその SQL クエリを追跡します。実行中の 4.22.1.0 インスタンスが 160 件のレコードを報告する理由を調査し、その後、Usage Sanity Check メーラーや Prometheus エクスポーターの出力を壊すことなく、False Positive の件数が解消されていることを確認します。

索引モデルが issue の本文から書いたものです。