Traefik dataplane never refreshes session.*.last_access for TCP and long-lived WebSocket circuits
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
In Traefik mode the network_timeout idle checker terminates live sessions. TCP circuits are affected from the first evaluation; WebSocket circuits are affected once the connection outlives max_network_inactivity_seconds without a new HTTP request.
== What the checker actually reads ==
NetworkTimeoutChecker._prepare_states (manager/sokovan/idle_check/checkers/network_timeout.py:133-138) reads exactly two keys per session:
- session..last_access, via ValkeyLiveClient.get_session_last_access_batch (common/clients/valkey_client/valkey_live/client.py:199-213)
- session..active_app_connections, via count_active_connections_batch
circuit..last_access is never read by any idle checker. And active_app_connections is written only by manager/services/stream/service.py (track_connection / untrack_connection / gc_stale_connections), which is the manager's own WebSocket stream proxy - the app proxy dataplane never writes it. So on every app-proxy path active_connections is 0 and session.\*.last_access is the only signal that keeps a session alive.
A missing key is not treated as activity: with last_access None, _decide_session_activity falls through to effective_expire_at = expire_at or (now + max_network_inactivity_seconds), persists it, and terminates on a later pass. A session whose marker is never written therefore always dies after max_network_inactivity_seconds.
== Native dataplane (correct) ==
BaseBackend.mark_last_used_time (appproxy/worker/proxy/backend/base.py:44-52) writes both session..last_access and circuit..last_access. It is driven by:
- plain HTTP: once per request (backend/http.py:184)
- WebSocket: LastAccessMarkerTask every 1.5s for the whole connection (backend/http.py:318, backend/last_access_marker.py:13)
- TCP: LastAccessMarkerTask every 1.5s for the whole connection (backend/tcp.py:83)
The per-connection cron is what keeps an idle-but-open TCP or WebSocket connection alive.
== Traefik dataplane (broken for TCP and long-lived WS) ==
HTTP is fine. The Go marker plugin (lablup/backend.ai-appproxy-worker-traefik, go/plugin.go:59-70) HEADs both circuit..last_access and session..last_access on every request, using the sessionids the coordinator injects at coordinator/models/circuit.py:478.
TCP is structurally impossible today. Circuit.traefik_routers (coordinator/models/circuit.py:339-366) attaches middlewares only on the HTTP branch; the TCP branch emits a router with no middlewares key at all, and get_traefik_middlewares (circuit.py:447) returns an empty mapping for anything but HTTP. Both plugins declare type: middleware in .traefik.yml, and Traefik TCP routers accept only tcp.middlewares (ipAllowList, inFlightConn), so no plugin can attach. The code already acknowledges this at circuit.py:302-304, where the client-IP allowlist had to be folded into the router's ClientIP match rule for the same reason. Result: neither key is ever written for a TCP circuit.
WebSocket over Traefik is marked once, at connection start. The Go plugin runs MarkLastUsed on request entry only; afterwards nothing refreshes the session key. _ActiveCircuitWriterTask (worker/proxy/frontend/traefik.py:34, 183-188) does run every 5s for circuits marked active, but it writes only circuit..last_access, which the checker does not read. So a WebSocket that stays open past max_network_inactivity_seconds without a new HTTP request is judged idle while fully connected.
== Fix direction ==
WebSocket half is small and self-contained: AbstractTraefikFrontend.active_circuit_writer should also refresh the session keys of each active circuit, not just the circuit key. The frontend already holds BaseFrontend.circuits (worker/proxy/frontend/base.py:19), so Circuit.session_ids is reachable; active_circuits is currently a set of circuit UUIDs and needs a lookup by circuit id.
TCP half needs a signal that does not depend on an HTTP middleware - Traefik access logs, TCP router metrics, or worker-side connection tracking. If that design turns out larger than expected, split the TCP half into its own issue and keep this one for the WebSocket fix.
Related: BA-6864 pins the plugin version. Note that the session key scheme was wrong before plugin commit ae6c6c6 (fix: invalid last_access key scheme, 2025-10-24) - it wrote session. without the .last_access suffix, so any worker running an older plugin build is additionally broken on the HTTP path.
== Reproduction ==
1. Run App Proxy in Traefik mode.
2. Create an interactive session with a network idle checker whose max_network_inactivity_seconds is short (e.g. 120).
3. Open a TCP app circuit (e.g. SSH) and keep the connection open and quiet.
4. Observe the session terminated after max_network_inactivity_seconds while the TCP connection is still established.
5. Repeat with a WebSocket app circuit, holding the socket open without issuing further HTTP requests: same outcome.
6. Repeat both on the native dataplane: the session survives.
## Success Criteria
- [ ] Traefik WebSocket: a circuit held open past max_network_inactivity_seconds with no further HTTP request keeps session..last_access advancing and is not terminated
- [ ] Traefik TCP: an established, quiet TCP circuit past max_network_inactivity_seconds is not terminated
- [ ] the refreshed keys are the session keys the checker reads, verified against ValkeyLiveClient.get_session_last_access_batch, not only circuit..last_access
- [ ] closing the connection stops the refresh, so the session becomes idle-eligible again within max_network_inactivity_seconds
- [ ] native dataplane behaviour is unchanged for HTTP, WebSocket and TCP
- [ ] regression test covers a Traefik circuit with no HTTP request activity across at least two idle-check evaluation passes
- [ ] pants test passes for affected packages
JIRA Issue: BA-7413
Contributor guide
Research direction
Start with AbstractTraefikFrontend.active_circuit_writer in worker/proxy/frontend/traefik.py and compare it with BaseBackend.mark_last_used_time and NetworkTimeoutChecker._prepare_states. Reproduce the issue, then add regression coverage for idle-check passes and connection closure; done means session.*.last_access refreshes for WebSocket and TCP circuits without HTTP activity and pants tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100