tests failed due to SIGTERM (some taking too long?)

This test failed on a CI run on pull request #9738:
https://github.com/oxidecomputer/omicron/pull/9738/checks?check_run_id=61803187859

Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01KG3BFCNEACHCGQZBPHMKZ5GF/sb6eF2wwA1gevOEiibwUt2Lf6LXZBTAS7DShNHJvvflfp5A3/01KG3BH06ANT7K2E5ETK9MPEG5#S8692

Excerpt from the log showing the failure:

  Cancelling due to signal: 6 tests still running
        PASS [  23.366s] (1829/2701) omicron-nexus::test_all integration_tests::updates::test_request_without_api_version
     SIGTERM [   7.853s] (1830/2701) omicron-nexus::test_all integration_tests::users_builtin::test_users_builtin
  stdout ───
    running 1 test
  stderr ───
    log file: /var/tmp/omicron_tmp/test_all-15ef63339c3c575c-test_users_builtin.6160.0.log
    note: configured to log to "/var/tmp/omicron_tmp/test_all-15ef63339c3c575c-test_users_builtin.6160.0.log"
    DB URL: postgresql://root@[::1]:40248/omicron?sslmode=disable
    DB address: [::1]:40248
    (test aborted with signal 15: SIGTERM)
     SIGTERM [   9.004s] (1831/2701) omicron-nexus::test_all integration_tests::updates::test_update_status
...
    (test aborted with signal 15: SIGTERM)
────────────
     Summary [7165.504s] 1834/2701 tests run: 1829 passed (101 slow), 5 failed, 6 skipped
     SIGTERM [   7.853s] (1830/2701) omicron-nexus::test_all integration_tests::users_builtin::test_users_builtin
     SIGTERM [   9.004s] (1831/2701) omicron-nexus::test_all integration_tests::updates::test_update_status
     SIGTERM [  53.152s] (1832/2701) omicron-nexus::test_all integration_tests::unauthorized::test_unauthorized
     SIGTERM [  11.969s] (1833/2701) omicron-nexus::test_all integration_tests::updates::test_trust_root_operations
     SIGTERM [  34.022s] (1834/2701) omicron-nexus::test_all integration_tests::updates::test_repo_upload
warning: 867/2701 tests were not run due to signal
error: test run failed

I thought maybe this was a buildomat thing but I see that @paudmir saw this. @smklein also saw something similar but most of the links in that issue are dead so I can't tell if this is a dup.

In #9377 we something similar. A comment from @sunshowers (generated by Trunk?) suggests this was due to a timeout or external intervention? Then a comment from me suggests that we do have a timeout that sends SIGTERM. Here's the timeout invocation:
https://github.com/oxidecomputer/omicron/blob/5472a8a045412a5f88bbb69881fa4a5966975058/.github/buildomat/build-and-test.sh#L136-L137

Looking more closely at my log, I do think that's what happened here because the tests started at 23:33:58, two hours before the SIGTERM. I have not yet dug into why this run took so much longer than usual.

Start with .github/buildomat/build-and-test.sh at the timeout invocation, then compare the linked CI log and the test_all summary with the related issues #9333, #4779, and #9377. Determine whether the two-hour timeout caused the SIGTERM and why this run took so long; done means the cause and an actionable fix are established.