Duplicate resource accounting update for a stopped/deleted instance

We have a project on rack2 that shows negative compute usage which manifested as an error during disk deletion:

{
  "msg": "request completed",
  "v": 0,
  "name": "65a11c18-7f59-41ac-b9e7-680627f996e7",
  "level": 30,
  "time": "2024-04-12T14:28:00.793206966Z",
  "hostname": "oxz_nexus_65a11c18-7f59-41ac-b9e7-680627f996e7",
  "pid": 10277,
  "uri": "//v1/disks/hubris-disk-lpc55xpresso-279c894b-ed57-4d14-8100-5928cf03924c?project=hubris-build",
  "method": "DELETE",
  "req_id": "aa0543d3-0c79-4faa-9b86-8ada4e0ca201",
  "remote_addr": "172.20.16.182:47642",
  "local_addr": "172.30.2.5:443",
  "component": "dropshot_external",
  "file": "/home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837",
  "error_message_external": "Internal Server Error",
  "error_message_internal": "saga ACTION error at node \"no_result1\": unexpected database error: value is too small for a byte count",
  "latency_us": 133503,
  "response_code": "500"
}

A database query again the virtual_resource_collection table for the related project showed that it had negative compute usage:

  a494b278-d167-4940-9ffb-6d900ea0e47d | 2024-04-12 14:28:00.75635+00  | Project         |                              0 |               -4 |     -8589934592

Here are the vmm/instances that were in the project previously:

root@[fd00:1122:3344:105::3]:32221/omicron> select id, name, ncpus, memory, state, time_created, time_deleted from instance where project_id = 'a494b278-d167-4940-9ffb-6d900ea0e47d' order by time_created;
                   id                  |                              name                               | ncpus |   memory   |   state   |         time_created          |         time_deleted
---------------------------------------+-----------------------------------------------------------------+-------+------------+-----------+-------------------------------+--------------------------------
  c306cd25-bc22-4b40-be8a-d1f7ed3850fd | hubris-build-oxide-rot-1-ad95a39e-de73-43f6-ad83-053fb8b91fad   |     4 | 8589934592 | destroyed | 2024-04-04 19:54:39.735178+00 | 2024-04-04 20:00:43.863439+00
  4709d422-537c-4021-8881-c3f2c9dc68fb | hubris-build-oxide-rot-1-dev-ad95a39e-de73-43f6-ad83-053fb8b91f |     4 | 8589934592 | destroyed | 2024-04-04 19:54:39.737293+00 | 2024-04-04 20:01:08.697802+00
  1d99250b-d425-4d93-bae1-9f87393cfaa4 | hubris-build-oxide-rot-1-50be9da6-43dd-48ed-b68f-591e3a969fbc   |     4 | 8589934592 | destroyed | 2024-04-04 20:03:43.664052+00 | 2024-04-04 20:10:40.70392+00
  b899cf15-547d-4d0a-ac04-2b137e85434e | hubris-build-oxide-rot-1-dev-50be9da6-43dd-48ed-b68f-591e3a969f |     4 | 8589934592 | destroyed | 2024-04-04 20:03:43.664272+00 | 2024-04-04 20:10:16.119842+00
  c25a7cf2-10ce-4c7a-b344-f8800734627c | hubris-build-oxide-rot-1-dev-4ac8baca-afec-4c0e-9782-3a42e32454 |     4 | 8589934592 | destroyed | 2024-04-04 20:20:07.852417+00 | 2024-04-04 20:26:30.777506+00
  02977cb7-180c-4bf7-af6f-3c2740c3f3d8 | hubris-build-oxide-rot-1-4ac8baca-afec-4c0e-9782-3a42e324542f   |     4 | 8589934592 | destroyed | 2024-04-04 20:20:07.857239+00 | 2024-04-04 20:26:21.589897+00
  51842add-0437-43a0-b8e4-24f6d936b694 | hubris-build-lpc55xpresso-cd98825f-d6db-40e0-b392-4198d68361e3  |     4 | 8589934592 | destroyed | 2024-04-11 21:14:39.701095+00 | 2024-04-11 21:18:24.44294+00
  951d3327-5d74-4934-82b9-921596296138 | hubris-build-oxide-rot-1-cd98825f-d6db-40e0-b392-4198d68361e3   |     4 | 8589934592 | destroyed | 2024-04-11 21:14:39.707547+00 | 2024-04-11 21:18:28.578122+00
  0b33f7f7-1629-4ba6-9a96-c40f3328f26a | hubris-build-rot-carrier-cd98825f-d6db-40e0-b392-4198d68361e3   |     4 | 8589934592 | destroyed | 2024-04-11 21:14:39.707748+00 | 2024-04-11 21:18:31.286399+00
  1f164a98-c0ef-4a3d-b773-7b4125d20512 | hubris-build-rot-carrier-fb64e4dc-9339-4c94-b96c-4082335fd518   |     4 | 8589934592 | destroyed | 2024-04-11 21:19:08.822619+00 | 2024-04-12 14:23:25.212172+00
  22ba56bb-e4d7-40f3-ad48-c497a19927fc | hubris-build-oxide-rot-1-fb64e4dc-9339-4c94-b96c-4082335fd518   |     4 | 8589934592 | destroyed | 2024-04-11 21:19:08.831067+00 | 2024-04-11 21:24:08.810187+00
  de9b28fb-329c-4dcb-84af-d433a78a4e16 | hubris-build-lpc55xpresso-fb64e4dc-9339-4c94-b96c-4082335fd518  |     4 | 8589934592 | destroyed | 2024-04-11 21:19:08.831157+00 | 2024-04-11 21:24:08.309922+00
  a1b7498a-af14-4e5b-8509-3c61ab3e3a34 | hubris-build-oxide-rot-1-1ebe6ce9-ee71-4f1a-bdc2-e87621887a3f   |     4 | 8589934592 | destroyed | 2024-04-12 14:12:23.851901+00 | 2024-04-12 14:18:28.260342+00
  5a2f39cc-2e64-4910-956c-c136dc38cb2f | hubris-build-oxide-rot-1-c2a05043-7b52-46b9-9a2b-84f591499979   |     4 | 8589934592 | destroyed | 2024-04-12 14:19:31.630662+00 | 2024-04-12 14:25:42.181182+00
  1d85d5e0-b38c-4550-8ba1-2988925a6927 | hubris-build-lpc55xpresso-279c894b-ed57-4d14-8100-5928cf03924c  |     4 | 8589934592 | destroyed | 2024-04-12 14:24:18.699901+00 | 2024-04-12 14:28:00.439503+00
  fe48a0be-d78b-4959-961d-92fa74d50bda | hubris-build-oxide-rot-1-279c894b-ed57-4d14-8100-5928cf03924c   |     4 | 8589934592 | destroyed | 2024-04-12 14:24:18.706574+00 | 2024-04-12 14:27:50.841866+00
  b3b42394-a7d9-4175-9c8c-75819408a328 | hubris-build-rot-carrier-279c894b-ed57-4d14-8100-5928cf03924c   |     4 | 8589934592 | destroyed | 2024-04-12 14:24:18.707483+00 | 2024-04-12 14:27:56.580457+00

Based on the deletion timestamps of the instances and disks (not included here), the most recent three deleted instances are likely the one that contributed to the usage accounting issues.

The nexus log showed that one of the instances went through resource clean up twice:

14:27:56.227Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): vmm is terminated, cleaning up resources
    file = nexus/src/app/instance.rs:1627
    instance_id = b3b42394-a7d9-4175-9c8c-75819408a328
    propolis_id = aab1f007-a6d2-4005-a4c3-f3673054344c 
...
14:27:56.264Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (ServerContext): vmm is terminated, cleaning up resources
    file = nexus/src/app/instance.rs:1627
    instance_id = b3b42394-a7d9-4175-9c8c-75819408a328
    propolis_id = aab1f007-a6d2-4005-a4c3-f3673054344c

The duplicate reduction in usage matches the negative 4 vcpus and 8 GB memory usage in the collection table.

Start at nexus/src/app/instance.rs:1627 and trace the resource-cleanup path for repeated "vmm is terminated, cleaning up resources" events. Compare its updates to virtual_resource_collection with the instance deletion sequence; done means cleanup does not decrement usage twice and the reported accounting failure is prevented or covered by a regression test.