lablup / lablup/backend.ai

Make the AppProxy traefik plugin repo public, pin its version, and verify its contract with core

Open
#12,816 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Decision: do NOT move the plugin source into the core monorepo

The original proposal was to fold the plugin source into `lablup/backend.ai`. That was **rejected** after investigation:

- The plugin is **not Backend.AI code**. It is _traefik_ code: the Rust half is executed by traefik's **WASM runtime**, the Go half is interpreted by traefik's **Yaegi** interpreter. Core never imports or executes it - it only writes the middleware config into etcd, and traefik loads the plugin from its own `plugins-local` directory.
- Integrating it would drag a **Rust toolchain +** `wasm32-wasip1` target into a Python monorepo, for an artifact core never runs.
- Pants (this distribution) ships **37 backends, none for Go or Rust**, so the build would be a `shell_command` shell-out hack.
- It would also add a new packaging burden: shipping the built `plugin.wasm` inside core's distribution so the installer can drop it into traefik's directory.

Integration was a _means_, not the goal. The two problems it was meant to solve are addressed far more cheaply below.

## Problem 1 - the plugin repo is private (the real blocker)

The installer fetches `appproxy-traefik-plugin.tar.gz` from a GitHub release of `lablup/backend.ai-appproxy-worker-traefik`. Because that repo is **private**, anonymous `curl` 404s, so the installer falls back to `gh release download`. That fallback **does not work for external users or for isla-sorna**, which is exactly why traefik-mode rollout was put on hold.

**Fix:** make the plugin repository public, then remove the `gh` fallback from the installer.

## Problem 2 - version drift between the plugin and core (already caused a bug)

The plugin has a **tight contract** with core: the marker unix-socket key scheme and the config fields passed through etcd. Because the repos evolve separately, they drifted:

| |key the plugin sends|key core's coordinator reads|result|
|---|---|---|---|
|deployed version (dogbowl)|`circuit.{id`}|`circuit.{id}.last_access`|miss|
|plugin repo `main`|`circuit.{id}.last_access`|same|hit|

The worker stores the key **verbatim** (`worker/proxy/frontend/traefik.py:163-167`); the coordinator reads `circuit.{id}.last_access` and, on a miss, **falls back to the circuit's** `created_at` and collects it once older than `unused_circuit_collection_timeout` (default 3600s), deleting the circuit and its etcd routes (`coordinator/server.py:645-691`).

The only other writer of `.last_access` is `active_circuit_writer`, fed by `mark-active`, which the plugin calls **only for websocket requests**. So circuits that never upgrade to a websocket (e.g. TensorBoard, MLflow, a plain HTTP app served from the container) can be collected an hour after creation even while in use. Jupyter / terminal / VSCode are websocket-based and therefore unaffected.

**Severity is low in practice** - dogbowl shows only ~9 circuit collections in 7 days - so this is a latent correctness defect, not an active outage.

**Fix:** pin the plugin version explicitly, upgrade the deployed plugin, and add a contract check in core so the key scheme cannot silently drift again.

## Scope

1. Make `lablup/backend.ai-appproxy-worker-traefik` public.
1. Remove the `gh release download` fallback from the installer; plain `curl` of the release asset is enough once public.
1. Pin the plugin version explicitly in core (constant already exists: `APPPROXY_TRAEFIK_PLUGIN_VERSION`) and bump it to a release built from current `main` (correct `.last_access` key scheme).
1. Before releasing that version, remove the leftover debug print in `go/plugin.go`: `os.Stderr.WriteString("########## New, config: %+v $$$$$$$$$$")` - it spams stderr once per circuit.
1. Add a contract test / documented invariant in core covering the marker key scheme (`circuit.{id}.last_access`, `session.{id}.last_access`) and the `mark-active` / `mark-inactive` UUID form.
1. Redeploy the plugin on dogbowl (wsw1 HTTP, wsw3 TCP) and clean up the dead `plugins-local/src/backend.ai/fallback-response-plugin/` directory (not declared in `localPlugins`; its `go.mod` module name duplicates the `-go` plugin).

## Acceptance criteria

- The plugin repo is public and the installer provisions the plugin with no `gh` fallback.
- The pinned plugin version writes `circuit.{id}.last_access`, verified in Valkey after deployment.
- A non-websocket HTTP app circuit survives past `unused_circuit_collection_timeout` while in use.
- Core has a check that fails if the marker key scheme drifts.

## Related

- BA-1982 / PR #12749 (TUI installer traefik mode) - unblocked by the public transition.
- isla-sorna traefik rollout - blocked on the same private-repo issue.

JIRA Issue: BA-6864

Contributor guide

Open the contributing guide

Research direction

Start by reading worker/proxy/frontend/traefik.py:163-167 and coordinator/server.py:645-691 to trace the marker-key contract, then inspect the installer’s plugin download logic and APPPROXY_TRAEFIK_PLUGIN_VERSION. Review go/plugin.go for the leftover debug output. Done means the public release installs without gh, the pinned version uses the documented keys, core has a contract check, and deployment validation confirms active non-websocket circuits survive.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python, rust, wasm
Domain
backend, devops, infrastructure, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.