livecycle / livecycle/preevy

Allow tunnel server upgrade without disconnecting user environments

Open
#233 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug enhancement need spec points: 5
Dominant language
TypeScript
Stars
2.2k
Forks
91
PR merge metrics
No merged PRs in 30d

Description

Currently, when deploying the tunnel server, user environments will be briefly disconnected while the CTA (agent) reconnects to the new instance. This can cause incoming requests to the environments to fail with 502 "environment not found" errors.

Suggested solution - a cooperative rollout flow, compatible with [Kubernetes rolling update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/) (although it's quite generic and can be used with other orchestration infra).

- The tunnel server will handle `SIGTERM` to start a graceful shutdown flow. It will notify its connected clients to reconnect (see below). It will then wait for all its client connections to end, or a configurable timeout has passed, then exit.
- When the CTA (agent) is notified of the pending tunnel server shutdown, it will:
- Create a new SSH client connection to its configured tunnel server URL. The new connection will be routed to the new tunnel server instance by the infra (e.g, K8s).
- Once the new SSH client connection is established, all existing forwards will be established on it. This will cause new requests to come in through the new SSH connection.
- Existing TCP forward connections from the old SSH connection will be allowed to complete. This assumes they are short-lived HTTP requests. Long-lived connections (e.g, websockets) will eventually be terminated from the remote side (the tunnel server timeout expiring), but are assumed to be designed to recover from disconnections.
- Once all the old TCP connections are closed, the old SSH connection will be closed.

Currently there is no simple way for the SSH server to notify its clients of an event. An applicative "server events" channel can be created by having the CTA initiate a specific "control" command session (exec) on its client connection, and wait for it to end as a signal. Alternatively, instead of using the SSH connection, the CTA can accept an HTTP request on its own API endpoint. However, this requires the tunnel server to identify the specific tunnel for each connected CTA.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the tunnel server shutdown handling and the CTA's SSH connection and forwarding flow. Review the Kubernetes rolling update behavior described in the issue, then resolve how server events will notify each CTA. Done means a server can receive SIGTERM, coordinate reconnection and forwarding migration, let existing connections finish or time out, and exit cleanly without losing new requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, typescript
Domain
devops, infrastructure, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.