Websocket message acknowledgement and maintenance

Current behavior

With @ozer550's awesome work on introducing websockets to Kolibri Studio, there's some opportunity for refinement on managing the communication with the backend over the websocket. Unfortunately, we didn't get a production-like server running until late in the project, so this proposal comes from testing in that environment as a post-project follow-up. This proposal would also reduce risk of critical bugs that could cripple editing functionality, and could serve as an ease of transition when we deploy websockets to production

• Currently, if the websocket gets closed, the frontend does not robustly handle it and requires you to refresh the page
• The connection could timeout from becoming idle
• The websocket message structure isn't standardized

Some insights:

• Google's Load Balancer timeout is 30 seconds so after 30 seconds, an idle websocket connection would be closed
• Cloudflare's timeout is 100 seconds, and would similarly close an idle connection

Proposal

• First, define a message structure with fields that can be used to provide integrity of the websocket message communication
  • An example lending some inspiration from TCP might be something like:

  {
    "id": "abc...123", // unique message id
    "type": "ACK", // type of message, perhaps SYN=synchronize, ACK=acknowledge, PNG=ping?
    "payload": { ... }, // existing request or response data
  }
  

• Define a timeout, 15 seconds? 25 seconds?

Desired behavior

• A new JS class, sync class, should be defined to abstract away the syncing of changes over HTTP API or Websockets
• The sync class can emit events when it receives changes, successes, allowed, disallowed, etc. and corresponding handle* functions can be attached as listeners
  • handleMaxRevs could be attached to the 3 events changes, successes, and errors
• The sync class should have a send method accepting a payload to send
• Every websocket message sent from the client should be ACKnowledged by the server
  • The server's response would still occur as a post-ack SYN message
  • The server would acknowledge with the message ID
• The client sends periodic PNG messages to keep connection from going idle
  • Send ping messages every timeout seconds
• If a ping message is not acknowledged after timeout seconds the connection
  • The sync class should handle fallback to the HTTP API with new messages
  • The sync class should re-ping on receiving new messages to send, up to two times (3 unacknowledged pings), while falling back to the HTTP API until an acknowledged ping is received
  • Pinging should always be timeout seconds after last acknowledged message
• The sync class should wait until receiving the response SYN message before SYNchronizing more changes
  • This wait time should be timeout seconds, meaning at most, a full request to response could wait up to 2 x timeout seconds, since the client would first receive the ACK message up to timeout seconds after sending
• Track to Sentry or Google Analytics each time a websocket fails, so that we can tune and simplify this in the future

Value add

• Stability as we roll out websockets to production
• Robust communication over websocket or HTTP API

Possible tradeoffs

• The message acknowledgement and associated work may be overkill, since we already have similar features represented in the change handling, but it does allow us to manage issues with the websocket connection separately from the changes and that architecture