Runner stops reporting in the middle of jobs

Describe the bug
We're seeing an issue on our self-hosted runners where jobs will stop having logs reported in the middle of a step. When this happens, the job continues (hanging on the step it was in the middle of), and cannot be manually cancelled. They will eventually be cancelled automatically by the job's timeout-minutes (if it has one). After they're cancelled, the logs of the step that the job timed out on (and any subsequent steps on the job), will be The log was not found. It may have been deleted based on retention settings.. The runner service itself also doesn't appear to be crashing/restarting when this happens.

I have noticed some Caught exception during append web console line to websocket, let's fallback to sending via non-websocket call (total calls: 49, failed calls: 1, websocket state: Open). errors in the _diag files for the runs, but I'm unsure whether or not this is normal. You can see them in the log files I uploaded below.

To Reproduce
Unfortunately I do not have good reproduction steps here. It seems to randomly happen on any step in any of the jobs that are sent to the runners on this machine.

Expected behavior
Jobs either pass for fail without hanging, and with all of their logs intact.

Runner Version and Platform

Version of your runner: 2.296.2

OS of the machine running the runner?: Mac Studio (M1 Ultra), Mac OS 12.3.1

What's not working?

Please include error messages and screenshots.

Job Log Output

This is what a hung job looks like, the output in the passed steps is normal so I'm not including it here. You'll notice this one hung on the clean up step, but the step seems totally random and I've seen it hang in any of the other steps as well:

Runner and Worker's Diagnostic Logs

The following is from the job in the screenshot above:

STDOUT/ERR from the relevant job/time period:

Current runner version: '2.296.2'
2022-09-19 05:01:48Z: Listening for Jobs
2022-09-19 17:29:56Z: Running job: Build Beta app
2022-09-19 18:34:43Z: Job Build Beta app completed with result: Canceled

_diag file: diag_file.txt

Please let me know if there's any other info we can provide.

Start with the attached diag_file.txt and the runner's _diag logs, focusing on the reported websocket append failures and the runner 2.296.2 behavior on Mac Studio with macOS 12.3.1. Reproduce the intermittent failure if possible; done means jobs no longer hang mid-step, remain cancellable, and retain logs after completion or cancellation.