Record all viewer crashes in metrics

What does this feature/enhancement do? Why is it needed?

(comment originally by kyle, edits by others based on changing problem definition)

Viewer crash metrics are potentially unreliable for multiple reasons, and are completely absent in the case of Gamecast/Project Zero. This proposal adds a new watchdog process mechanism and a new server endpoint for crash data collection. Under the new system, crashes will be reported as soon as a viewer process exits.

Note: there are also issues with bugsplat reports not getting sent in many cases, but these will be considered later. First step is to get reliable metrics.

Current Crash Metrics System

• Viewer accumulates various marker files based on progress during login, normal operation, and crash or shutdown
• If viewer crashes, then the next viewer session will review the marker files to generate last_exec_event and last_exec_duration. This info is sent to login and from there to SI. The last_exerc info is stored with the session information for the later viewer session (not directly associated with the session that crashed)

Problems with this approach

• If there are multiple crashes before login is contacted again, some crash data is lost
• If there is no 2nd viewer session on that machine, the crash data is never sent
• Project Zero/Gamecast: If the host gets reset after every session (as happens with a streaming service), the crash data is never sent
• There is no completely reliable way to associate crash data with the session that actually crashed
• It may be possible to double count crashes if the 2nd viewer session terminates abnormally between sending the last_exec info and clearing the marker files

Proposed alternative

• Every viewer session starts a watchdog process
• If the viewer exits abnormally, watchdog process sends data to a new endpoint in the login service. The data sent includes crash cause, session duration, session id if known, viewer version
• Watchdog then shuts down

Test Plan

Include information, such as guidance and test scenarios, QA will need to test this feature.

Test the following scenarios and verify that either a correctly labeled crash is logged with SI, or a successful logout record is generated:

• Viewer forced crash via menu (various causes)
• Using a hacked viewer, crash automatically at various early stages, including before login is contacted
• Viewer session terminated by laptop being closed
• Viewer session terminated by user logout
• Viewer session terminated by client system being shut down (laptop/desktop)
• (Gamecast) Viewer session terminated by tab being closed
• Normal logout via the file menu
• Normal logout by closing the viewer window

Verify that the watchdog process terminates normally in all these cases

Start by tracing the current marker-file flow through login and SI, then review the proposed watchdog process and new login-service endpoint. Done means crash and successful logout records are correctly logged across the listed viewer and Gamecast scenarios, with the watchdog terminating normally in each case.