lablup / lablup/backend.ai

Traefik dataplane never refreshes session.*.last_access for TCP and long-lived WebSocket circuits

Open
#13,854 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

In Traefik mode the network_timeout idle checker terminates live sessions. TCP circuits are affected from the first evaluation; WebSocket circuits are affected once the connection outlives max_network_inactivity_seconds without a new HTTP request.

== What the checker actually reads ==

NetworkTimeoutChecker._prepare_states (manager/sokovan/idle_check/checkers/network_timeout.py:133-138) reads exactly two keys per session:
- session..last_access, via ValkeyLiveClient.get_session_last_access_batch (common/clients/valkey_client/valkey_live/client.py:199-213)
- session..active_app_connections, via count_active_connections_batch

circuit..last_access is never read by any idle checker. And active_app_connections is written only by manager/services/stream/service.py (track_connection / untrack_connection / gc_stale_connections), which is the manager's own WebSocket stream proxy - the app proxy dataplane never writes it. So on every app-proxy path active_connections is 0 and session.\*.last_access is the only signal that keeps a session alive.

A missing key is not treated as activity: with last_access None, _decide_session_activity falls through to effective_expire_at = expire_at or (now + max_network_inactivity_seconds), persists it, and terminates on a later pass. A session whose marker is never written therefore always dies after max_network_inactivity_seconds.

== Native dataplane (correct) ==

BaseBackend.mark_last_used_time (appproxy/worker/proxy/backend/base.py:44-52) writes both session..last_access and circuit..last_access. It is driven by:
- plain HTTP: once per request (backend/http.py:184)
- WebSocket: LastAccessMarkerTask every 1.5s for the whole connection (backend/http.py:318, backend/last_access_marker.py:13)
- TCP: LastAccessMarkerTask every 1.5s for the whole connection (backend/tcp.py:83)

The per-connection cron is what keeps an idle-but-open TCP or WebSocket connection alive.

== Traefik dataplane (broken for TCP and long-lived WS) ==

HTTP is fine. The Go marker plugin (lablup/backend.ai-appproxy-worker-traefik, go/plugin.go:59-70) HEADs both circuit..last_access and session..last_access on every request, using the sessionids the coordinator injects at coordinator/models/circuit.py:478.

TCP is structurally impossible today. Circuit.traefik_routers (coordinator/models/circuit.py:339-366) attaches middlewares only on the HTTP branch; the TCP branch emits a router with no middlewares key at all, and get_traefik_middlewares (circuit.py:447) returns an empty mapping for anything but HTTP. Both plugins declare type: middleware in .traefik.yml, and Traefik TCP routers accept only tcp.middlewares (ipAllowList, inFlightConn), so no plugin can attach. The code already acknowledges this at circuit.py:302-304, where the client-IP allowlist had to be folded into the router's ClientIP match rule for the same reason. Result: neither key is ever written for a TCP circuit.

WebSocket over Traefik is marked once, at connection start. The Go plugin runs MarkLastUsed on request entry only; afterwards nothing refreshes the session key. _ActiveCircuitWriterTask (worker/proxy/frontend/traefik.py:34, 183-188) does run every 5s for circuits marked active, but it writes only circuit..last_access, which the checker does not read. So a WebSocket that stays open past max_network_inactivity_seconds without a new HTTP request is judged idle while fully connected.

== Fix direction ==

WebSocket half is small and self-contained: AbstractTraefikFrontend.active_circuit_writer should also refresh the session keys of each active circuit, not just the circuit key. The frontend already holds BaseFrontend.circuits (worker/proxy/frontend/base.py:19), so Circuit.session_ids is reachable; active_circuits is currently a set of circuit UUIDs and needs a lookup by circuit id.

TCP half needs a signal that does not depend on an HTTP middleware - Traefik access logs, TCP router metrics, or worker-side connection tracking. If that design turns out larger than expected, split the TCP half into its own issue and keep this one for the WebSocket fix.

Related: BA-6864 pins the plugin version. Note that the session key scheme was wrong before plugin commit ae6c6c6 (fix: invalid last_access key scheme, 2025-10-24) - it wrote session. without the .last_access suffix, so any worker running an older plugin build is additionally broken on the HTTP path.

== Reproduction ==

1. Run App Proxy in Traefik mode.
2. Create an interactive session with a network idle checker whose max_network_inactivity_seconds is short (e.g. 120).
3. Open a TCP app circuit (e.g. SSH) and keep the connection open and quiet.
4. Observe the session terminated after max_network_inactivity_seconds while the TCP connection is still established.
5. Repeat with a WebSocket app circuit, holding the socket open without issuing further HTTP requests: same outcome.
6. Repeat both on the native dataplane: the session survives.

## Success Criteria
- [ ] Traefik WebSocket: a circuit held open past max_network_inactivity_seconds with no further HTTP request keeps session..last_access advancing and is not terminated
- [ ] Traefik TCP: an established, quiet TCP circuit past max_network_inactivity_seconds is not terminated
- [ ] the refreshed keys are the session keys the checker reads, verified against ValkeyLiveClient.get_session_last_access_batch, not only circuit..last_access
- [ ] closing the connection stops the refresh, so the session becomes idle-eligible again within max_network_inactivity_seconds
- [ ] native dataplane behaviour is unchanged for HTTP, WebSocket and TCP
- [ ] regression test covers a Traefik circuit with no HTTP request activity across at least two idle-check evaluation passes
- [ ] pants test passes for affected packages

JIRA Issue: BA-7413

Contributor guide

Open the contributing guide

Research direction

Start with AbstractTraefikFrontend.active_circuit_writer in worker/proxy/frontend/traefik.py and compare it with BaseBackend.mark_last_used_time and NetworkTimeoutChecker._prepare_states. Reproduce the issue, then add regression coverage for idle-check passes and connection closure; done means session.*.last_access refreshes for WebSocket and TCP circuits without HTTP activity and pants tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.