A test was not retriggered when it should have been

In https://bugzilla.mozilla.org/show_bug.cgi?id=1703714 we retriggered 3 of the 4 failures (2 of 3 jobs). This happened in task cO5wL2X3RkWWy2K_o2a9sQ:

[task 2021-04-08T06:06:21.213Z] [Debug]   TaskclusterProvider: ================================================
[task 2021-04-08T06:06:21.213Z] [Info]    TaskclusterProvider: Beginning TaskclusterProvider.determine_jobs_to_retrigger
[task 2021-04-08T06:06:21.213Z] [Debug]   TaskclusterProvider:  Arguments: [Omitted]
[task 2021-04-08T06:06:21.213Z] [Debug]   TaskclusterProvider: failed_jobs_task_ids: {'MnvebJxcSmGSJy2_xOyORw', 'M4zTifrySZKdTAXissH48A', 'BJUwNkPtRW6irllKpJ7Nww'}
[task 2021-04-08T06:06:21.213Z] [Debug]   TaskclusterProvider: failed_jobs_with_health_classifications_task_ids: {'BJUwNkPtRW6irllKpJ7Nww', 'M4zTifrySZKdTAXissH48A', 'MnvebJxcSmGSJy2_xOyORw'}
[task 2021-04-08T06:06:21.213Z] [Info]    TaskclusterProvider: Requesting failure classifications
[task 2021-04-08T06:06:21.578Z] [Debug]   TaskclusterProvider: failed_jobs_with_taskcluster_classification_task_ids: set()
[task 2021-04-08T06:06:21.578Z] [Debug]   TaskclusterProvider: jobs_failed_with_no_health_classification_task_ids: set()
[task 2021-04-08T06:06:21.578Z] [Info]    TaskclusterProvider: Ending TaskclusterProvider.determine_jobs_to_retrigger
[task 2021-04-08T06:06:21.578Z] [Info]    Ending VendorTaskRunner._get_comments_on_push
[task 2021-04-08T06:06:21.578Z] [Info]    All jobs completed, we found failures we need to retrigger, going to retrigger 2 jobs: 
[task 2021-04-08T06:06:21.578Z] [Debug]   test-linux1804-64-qr/debug-xpcshell-e10s-5 M4zTifrySZKdTAXissH48A
[task 2021-04-08T06:06:21.578Z] [Debug]   test-android-em-7.0-x86_64-qr/debug-geckoview-mochitest-plain-e10s-1 MnvebJxcSmGSJy2_xOyORw

Why didn't we retrigger the other one? We should have only retriggered jobs if they were in jobs_failed_with_no_health_classification (which we can see from above they weren't) or they were in need_investigation_by_test. However by the time the retriggers were done running and Updatebot provided a summary they weren't needs investigation they were known issues.

My best guess here is that in the six hours between updatebot executions, those failures switched from needs investigation to known issues. And that the one failure (the one we didn't retrigger) was always a known issue.

If there was a way to query that status change from Push Health we could know for sure, but this is not particurally important.

It might be important to be able to come back to a bug and later and say "Actually these jobs that I thought Needed Investigation actually are Known Issues." So querying Push Health for a particular test status later would be more useful.

Start in apis/taskcluster.py around lines 173-189 and trace TaskclusterProvider.determine_jobs_to_retrigger using the logged task IDs and classification sets. Compare the retrigger decision with the later Updatebot summary, then investigate whether Push Health can provide the historical test status needed to explain the discrepancy.