Make the AppProxy traefik plugin repo public, pin its version, and verify its contract with core
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Decision: do NOT move the plugin source into the core monorepo
The original proposal was to fold the plugin source into `lablup/backend.ai`. That was **rejected** after investigation:
- The plugin is **not Backend.AI code**. It is _traefik_ code: the Rust half is executed by traefik's **WASM runtime**, the Go half is interpreted by traefik's **Yaegi** interpreter. Core never imports or executes it - it only writes the middleware config into etcd, and traefik loads the plugin from its own `plugins-local` directory.
- Integrating it would drag a **Rust toolchain +** `wasm32-wasip1` target into a Python monorepo, for an artifact core never runs.
- Pants (this distribution) ships **37 backends, none for Go or Rust**, so the build would be a `shell_command` shell-out hack.
- It would also add a new packaging burden: shipping the built `plugin.wasm` inside core's distribution so the installer can drop it into traefik's directory.
Integration was a _means_, not the goal. The two problems it was meant to solve are addressed far more cheaply below.
## Problem 1 - the plugin repo is private (the real blocker)
The installer fetches `appproxy-traefik-plugin.tar.gz` from a GitHub release of `lablup/backend.ai-appproxy-worker-traefik`. Because that repo is **private**, anonymous `curl` 404s, so the installer falls back to `gh release download`. That fallback **does not work for external users or for isla-sorna**, which is exactly why traefik-mode rollout was put on hold.
**Fix:** make the plugin repository public, then remove the `gh` fallback from the installer.
## Problem 2 - version drift between the plugin and core (already caused a bug)
The plugin has a **tight contract** with core: the marker unix-socket key scheme and the config fields passed through etcd. Because the repos evolve separately, they drifted:
| |key the plugin sends|key core's coordinator reads|result|
|---|---|---|---|
|deployed version (dogbowl)|`circuit.{id`}|`circuit.{id}.last_access`|miss|
|plugin repo `main`|`circuit.{id}.last_access`|same|hit|
The worker stores the key **verbatim** (`worker/proxy/frontend/traefik.py:163-167`); the coordinator reads `circuit.{id}.last_access` and, on a miss, **falls back to the circuit's** `created_at` and collects it once older than `unused_circuit_collection_timeout` (default 3600s), deleting the circuit and its etcd routes (`coordinator/server.py:645-691`).
The only other writer of `.last_access` is `active_circuit_writer`, fed by `mark-active`, which the plugin calls **only for websocket requests**. So circuits that never upgrade to a websocket (e.g. TensorBoard, MLflow, a plain HTTP app served from the container) can be collected an hour after creation even while in use. Jupyter / terminal / VSCode are websocket-based and therefore unaffected.
**Severity is low in practice** - dogbowl shows only ~9 circuit collections in 7 days - so this is a latent correctness defect, not an active outage.
**Fix:** pin the plugin version explicitly, upgrade the deployed plugin, and add a contract check in core so the key scheme cannot silently drift again.
## Scope
1. Make `lablup/backend.ai-appproxy-worker-traefik` public.
1. Remove the `gh release download` fallback from the installer; plain `curl` of the release asset is enough once public.
1. Pin the plugin version explicitly in core (constant already exists: `APPPROXY_TRAEFIK_PLUGIN_VERSION`) and bump it to a release built from current `main` (correct `.last_access` key scheme).
1. Before releasing that version, remove the leftover debug print in `go/plugin.go`: `os.Stderr.WriteString("########## New, config: %+v $$$$$$$$$$")` - it spams stderr once per circuit.
1. Add a contract test / documented invariant in core covering the marker key scheme (`circuit.{id}.last_access`, `session.{id}.last_access`) and the `mark-active` / `mark-inactive` UUID form.
1. Redeploy the plugin on dogbowl (wsw1 HTTP, wsw3 TCP) and clean up the dead `plugins-local/src/backend.ai/fallback-response-plugin/` directory (not declared in `localPlugins`; its `go.mod` module name duplicates the `-go` plugin).
## Acceptance criteria
- The plugin repo is public and the installer provisions the plugin with no `gh` fallback.
- The pinned plugin version writes `circuit.{id}.last_access`, verified in Valkey after deployment.
- A non-websocket HTTP app circuit survives past `unused_circuit_collection_timeout` while in use.
- Core has a check that fails if the marker key scheme drifts.
## Related
- BA-1982 / PR #12749 (TUI installer traefik mode) - unblocked by the public transition.
- isla-sorna traefik rollout - blocked on the same private-repo issue.
JIRA Issue: BA-6864
Contributor guide
Research direction
Start by reading worker/proxy/frontend/traefik.py:163-167 and coordinator/server.py:645-691 to trace the marker-key contract, then inspect the installer’s plugin download logic and APPPROXY_TRAEFIK_PLUGIN_VERSION. Review go/plugin.go for the leftover debug output. Done means the public release installs without gh, the pinned version uses the documented keys, core has a contract check, and deployment validation confirms active non-websocket circuits survive.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python, rust, wasm
- Domain
- backend, devops, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100