Self-contained demo loop: local environment + runbook for the multi-hour optimization session

# Self-contained demo loop: local environment + runbook for the multi-hour optimization session

## What this is

A fully local Accelerate environment — WP on Docker, ClickHouse, the flux traffic generator, and the AI toolkit — wired together to run the full A/B optimization loop end-to-end: create test → stream traffic → bandit converges → declare winner → iterate → learn. The goal is to drive the toolkit through enough real loops to surface behavioral gaps in the ability layer, the bandit pipeline, and the toolkit's reasoning. Everything below was proven end-to-end during a multi-phase validation run (30-day backfill, 39k+ PV/hr, four toolkit-created bandit experiments; one loop completed fully — create → start → bandit weight shifts → P2BB ≥ 0.95 → confirmatory → winner (+23% lift) → winner applied and serving — plus a learn-pass journal written per the toolkit spec). This issue is the recipe to run the longer session. Flux and the visualizer are test harness only — not part of the product loop and not being proposed for the plugin.

---

## Environment recipe

Steps are ordered. Each must succeed before the next.

> **`wp` in this document** means wp-cli against the Docker stack. Define once:
> ```bash
> alias wp='docker run --rm --volumes-from accelerate-wordpress-1 --env-file .wp-env --network accelerate_default wordpress:cli wp'
> ```
> (run from `~/dev/altis-accelerate` so `.wp-env` resolves)

### 1. Stack

```bash
cd ~/dev/altis-accelerate
git checkout fix/bandit-live-results # carries A2, A3, A9 fixes — demo won't work on main
composer start
```

Confirm WP is up at http://localhost:8081.

### 2. ClickHouse materialized views

The default `ch-setup` step only creates the base events table. The MVs (including the engaged-time MV) must be created separately:

```bash
docker exec accelerate-clickhouse-1 clickhouse-client \
--queries-file /.setup/create-analytics-materialized-views.sql
```

If the file isn't mounted inside the container, copy it first:

```bash
docker cp .setup/create-analytics-materialized-views.sql \
accelerate-clickhouse-1:/.setup/create-analytics-materialized-views.sql
```

### 3. WordPress setup

```bash
# WP 6.9+ required for abilities; 7.x confirmed working
wp core update

# Enable abilities API
wp option update accelerate_abilities_api_enabled 1

# Install the MCP adapter (not bundled in dev builds).
# Host paths aren't visible inside the wp-cli container, so extract the
# release zip straight into the WP volume, then activate:
gh release download --repo WordPress/mcp-adapter --pattern mcp-adapter.zip -D /tmp --clobber
unzip -oq /tmp/mcp-adapter.zip -d ~/dev/altis-accelerate/.wordpress/wp-content/plugins/
wp plugin activate mcp-adapter

# Create an admin Application Password for the toolkit
wp user application-password create 1 "accelerate-toolkit" --porcelain
# → note the returned password for /accelerate-connect
```

### 4. Flux — docker-compose.override.yml

Flux is not in the main compose file. It is mounted via an **untracked** `docker-compose.override.yml` in `~/dev/altis-accelerate/`. That file:

- Mounts `~/dev/accelerate-flux` as a WP plugin inside the container
- Mounts `~/dev/accelerate-flux/docker/clickhouse-async-inserts.xml` into ClickHouse `users.d/` (critical: prevents memory ceiling at demo rates — see C3)
- Sets `restart: unless-stopped` on the tracker container (prevents silent death on ClickHouse errors — see C2)

If you don't have this file, create it:

```yaml
# docker-compose.override.yml (do not commit — use ABSOLUTE paths;
# compose does not reliably expand ~)
services:
wordpress:
volumes:
- /Users//dev/accelerate-flux:/var/www/html/wp-content/plugins/accelerate-flux
clickhouse:
volumes:
- /Users//dev/accelerate-flux/docker/clickhouse-async-inserts.xml:/etc/clickhouse-server/users.d/async-inserts.xml:ro
tracker:
restart: unless-stopped
```

Then restart the stack:

```bash
composer stop && composer start
wp plugin activate accelerate-flux
```

### 5. Seed

Creates the ClearSpring Foundation demo site (settings, authors, posts, patterns, audiences, personalization, blocks). Idempotent — safe to re-run.

```bash
cd ~/dev/accelerate-flux
./seed/run.sh
```

### 6. Cron driver — fast-loop mode

Forces `altis_post_ab_test_cron` (normally hourly) every minute so bandit phases progress at demo speed:

```bash
cd ~/dev/accelerate-flux
nohup ./bin/cron-driver.sh >> /tmp/flux-cron-driver.log 2>&1 &
```

> To revert to realistic hourly cadence: comment out the second `docker run` block inside the `while` loop in `bin/cron-driver.sh`.

### 7. Fast-loop bandit thresholds (mu-plugin)

Burn-in and confirmatory sample sizes default to 50/100 (production). For rapid iteration use a mu-plugin to reduce them:

```php
15 );
add_filter( 'altis.experiments.bandit.confirmatory_min_conversions', fn() => 25 );
```

(The P2BB ≥ 0.95 significance bar is intentionally NOT filterable — close races take the time they take.)

**Delete this file** before any production-realistic session.

### 8. Traffic

Run backfill first (gives the dashboard 30 days of organic history), then start the stream:

```bash
# Historical ramp — 30 days, peak 600 PV/hr; checkpointed/resumable
wp flux backfill --days=30 --end-rate=600

# Live stream — 30k PV/hr proven stable; dial up to 40k+ if the stack is healthy
wp flux stream --pv-per-hour=30000 --flat
```

Confirm delivery is clean:

```bash
wp flux status
```

### 9. Toolkit connection

```
/accelerate-connect → http://localhost:8081
```

Use the Application Password from step 3. Then:

```
/accelerate-status
```

Must return green across all checks before proceeding.

### 10. Visualizer (optional — observer only)

Journals every run; useful for watching bandit weights shift in real time and replaying sessions.

```bash
cd ~/dev/accelerate-flux-ai-viz
npm run server # listens on :4317
```

Replay a past run:

```bash
npm run replay -- runs/ --speed=10
```

---

## The loop sequence

One iteration, repeated for each block under test:

| Step | Command / action | Notes |
|------|-----------------|-------|
| 1. Survey | `/accelerate-opportunities` or `/accelerate-review` | Identify which CTA block to test next |
| 2. Plan | `accelerate-test` → plan | Toolkit proposes variant copy |
| 3. Create | `accelerate-test` → create-ab-test | Toolkit calls the ability |
| **3a. Start (gap A1)** | `wp eval-file ~/dev/accelerate-flux/seed/start-experiment.php ` | **Required until A1 is fixed in the ability layer.** `create-ab-test` leaves the experiment paused; this script replicates the editor's start path (syncs variants, sets start meta, clears paused flag, inits bandit mode + burn_in phase). |
| 4. Wait for pickup | `wp flux status` | Flux re-discovers the test within ~60 s |
| 5. Monitor | `get-experiment-results` (toolkit) | Live results visible during bandit phases on this branch (A3 fix) |
| 6. Winner | automatic (bandit pipeline) | The analyser declares + completes at confirmatory budget. **Gap A13**: the winner is never APPLIED — the block keeps serving all variants 50/50, and `stop-experiment` refuses completed tests. Until fixed: rewrite the block content to the winning variant's inner markup (see A13 note), which also mints a ghost experiment row (A14). |
| 7. Push winner | `update-variant` or `add-variant` (toolkit) | Replace control with the winning copy |
| 8. Repeat | Back to step 1 for the next block | |
| ≥3 loops | `/accelerate-learn` | Produces the learning journal — patterns across all decisive tests |

---

## Known gaps & ground rules

Full findings ledger: `FINDINGS.md` in the (local, unpublished) viz harness repo `~/dev/accelerate-flux-ai-viz`. Summary inlined below — this issue is self-contained:

### Accelerate plugin (`altis-accelerate`)

| ID | Sev | Finding | Status |
|----|-----|---------|--------|
| A1 | 🔴 | `create-ab-test` / `create-personalization-rule` leave the experiment paused — no start metas, no variant sync. The toolkit would report "live" while nothing collects. | Workaround: `seed/start-experiment.php`; real fix in the ability |
| A13 | 🔴 | Bandit auto-completion never applies the winner — block serves all variants 50/50 after `done`; `stop-experiment` refuses completed tests, so the toolkit can't execute the ability layer's own recommendation. | Open — loop exit is broken like A1 breaks entry |
| A14 | 🟡 | Applying the winner by content rewrite (the only workaround) creates a spurious completed-no-winner experiment row that pollutes history and the learning journal. | Open |
| A2 | 🔴 | `get-experiment-results` inverted the Bayesian win probability (reported ~30% winner as 70%). Wrong winner recommendations. | **Fixed** on branch |
| A3 | 🔴 | Results stay empty during bandit phases — ability and UI were blind until completion. | **Fixed** on branch |
| A4 | 🟠 | Personalization results always zero via the ability even with data present. | Open |
| A5 | 🟠 | No ability reads a block's raw content; `create-ab-test` requires the control's content it can't supply. | Open |
| A6 | 🟠 | `broadcast-content` writes to an unregistered post type — can't render or query by type. | Open |
| A7 | 🟠 | Bandit phase-init gate is fragile: skips init if `test_mode` is pre-set. | Open |
| A8 | 🟡 | Broken JSON escaping in variant titles from `create-personalization-rule`. | Open (demo data hand-fixed) |
| A9 | 🟡 | `.setup/create-analytics-materialized-views.sql` column-count mismatch on clean installs aborts mid-file. | **Fixed** on branch |
| A11 | 🟡 | Ability permission callbacks reject user `0` — WP-CLI invocations need `--user`. | Documented |
| A12 | 🟡 | MCP adapter not bundled in dev builds; compose pins WP 6.7 vs 6.9+ requirement. | Documented |

### AI toolkit (`accelerate-ai-toolkit`)

| ID | Sev | Finding | Status |
|----|-----|---------|--------|
| B1 | 🔴 | `accelerate-test` verified only variant content post-create, so announced "live" for a paused test. | **Fixed** on `release/seamless-demo` |
| B2 | 🟠 | Backup-before-mutation referenced content sources that can't work for fresh blocks (A5 dependency). | **Fixed** on `release/seamless-demo` |
| B3 | 🟠 | No autonomous loop mode — every mutation gated on per-action confirmation. Correct for production; an auto-research-loop paradigm needs an explicit opt-in. | Roadmap observation |
| B5 | 🟡 | `accelerate-learn` spec: `patterns_with_signal` undefined, `last_updated` semantics unspecified — both hit on first real journal run. | **Fixed** on `release/seamless-demo` |
| B4 | 🟡 | Experiments list REST omits phase/weights/P2BB; variant titles not first-class in list payloads. | Viz works around it |

### Stack / infrastructure

| ID | Sev | Finding | Status |
|----|-----|---------|--------|
| C1 | 🔴 | Tracker ingests exactly one visitor per POST (silent drop of the rest). | Flux reshaped to 1/POST; upstream image needs fix |
| C2 | 🔴 | Tracker dies on ClickHouse errors with no restart — stayed dead through 144k failed deliveries. | Restart policy in override |
| C3 | 🟠 | ClickHouse memory ceiling at per-visitor INSERT storm. | Async-insert profile in `docker/` |
| C4 | 🟠 | Flux backfill reported success through mass delivery failure. | **Fixed** in flux |
| C5 | 🟠 | Long-running flux daemon never saw new tests (stale WP object cache). | **Fixed** in flux |
| C6 | 🟡 | WP went unresponsive under combined load once. | Observed; monitor during long session |
| C7 | 🟡 | Async-insert profile causes ClickHouse insert-then-read races in PHPUnit on this stack only. CI unaffected. | Known tradeoff |

### Ground rules for the session

- **Never modify the toolkit mid-session.** Run one branch end-to-end.
- **Log new gaps as comments on this issue** (ID, severity, one-line description).
- **Pages and posts are out of scope.** The loop is block-centric only.
- **The latent engine is the hidden ground truth.** Winners must be earned — the engine resisted a deliberately "strong" urgency variant (it lost to its calm control). Don't game it; the point is to observe whether the toolkit's optimization converges on the real signal.

---

## Remote-readiness checklist (product scope, not demo scope)

For this to be a shipped feature rather than a local harness:

- [ ] Merge `fix/bandit-live-results` (ships A2, A3, A9 fixes)
- [ ] Fix A1 in the ability layer: write start metas + sync variants + clear paused flag on `create-ab-test`, or add a `start-experiment` ability
- [ ] Fix A5: expose block raw content via an ability so `create-ab-test` can supply its own prerequisite
- [ ] Fix A13 (apply winner on bandit completion, or let stop-experiment act on completed tests) and A14
- [ ] Fix A4 and A7
- [ ] Cut toolkit release from `release/seamless-demo`

Flux and the visualizer are harness-only and stay out of the product. The visualizer may later become a separate experimental repo consuming real data via REST/abilities instead of direct ClickHouse.

---

## Success criteria for the multi-hour session

| Criterion | Target |
|-----------|--------|
| Completed loops | ≥ 5 end-to-end (create → stream → converge → declare → iterate) |
| Learning journal | Patterns present for ≥ 3 decisive tests via `/accelerate-learn` |
| Conversion lift | Iterated blocks show measurable CTR improvement vs original latent CTRs |
| Breakage | Zero unexplained failures (track anything unexpected as a comment) |

Start in the altis-accelerate checkout with composer start, then create the ClickHouse materialized views from .setup/create-analytics-materialized-views.sql and follow the numbered environment recipe. Use the mentioned flux scripts, WP-CLI commands, and toolkit entry points to run the loop; done means status checks stay green, multiple experiments complete, and /accelerate-learn produces the learning journal.

Written by the indexing model from the issue text.