MemberJunction / MemberJunction/MJ

[Computer Use] Test-variable guard has no escape for literal {{ }} in goal prose — aborts tests that describe MJ template syntax

Open
#3,319 0 comments 0 reactions 0 assignees View on GitHub
bug priority: medium
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Summary

The Computer Use test driver's fail-fast guard for unresolved `{{var}}` placeholders (CU-F7) has no escape hatch, so a test whose **goal prose legitimately contains mustache braces** aborts before the browser ever launches.

Real occurrence — regression test `T019 - Communication Templates`, whose goal describes MJ Template syntax to the agent:

> …a 'Contents' section with the template Text (which may include `{{ placeholder }}` parameters)…

```
Test execution failed: Unresolved test variable(s) after substitution: goal:{{placeholder}}.
Define them in the suite/test variables (or MJ_TEST_VAR_* env) before running.
```

The test fails in **33 ms** with score 0 and never opens a browser. It is retried once and fails identically.

## Why this is awkward

The class of tests most likely to need braces in the goal is exactly the class that tests MJ's own templating features (Communication Templates, AI Prompt templates, Jinja/Nunjucks editors). Today the only workaround is to avoid the notation when describing the notation.

## Affected code

- Guard: [`packages/AI/MJComputerUse/src/test-driver/ComputerUseTestDriver.ts`](https://github.com/MemberJunction/MJ/blob/next/packages/AI/MJComputerUse/src/test-driver/ComputerUseTestDriver.ts) ~L205–219
- Matcher: [`packages/AI/MJComputerUse/src/utils/variable-substitution.ts`](https://github.com/MemberJunction/MJ/blob/next/packages/AI/MJComputerUse/src/utils/variable-substitution.ts) L17–18

```typescript
const WHOLE_STRING = /^\{\{\s*([a-zA-Z_][\w.-]*)\s*\}\}$/;
const EMBEDDED = /\{\{\s*([a-zA-Z_][\w.-]*)\s*\}\}/g;
```

`{{ placeholder }}` matches `EMBEDDED` and is indistinguishable from a real variable reference.

## The guard itself is worth keeping

CU-F7 exists for a good reason and should not be relaxed wholesale: a literal `{{baseUrl}}` surviving into `startUrl` used to become a confusing navigation error ~30s into a run, with no hint that a suite variable was simply never provided. Failing fast and naming the key is the right behavior.

## Suggested fix direction

An explicit escape is probably the smallest correct change — e.g. treat `\{{ … }}` (or `{{{ … }}}`) as a literal, unescaping it during substitution and skipping it in `findUnresolvedPlaceholders`. That keeps the guard strict for genuine typos while letting authors write about mustache syntax.

An alternative — only treating a placeholder as a variable when its key is declared in the suite/test variable schema — is less appealing, since it would silently re-open the exact failure mode CU-F7 was added to close (an undeclared `{{baseUrl}}` would once again sail through).

## Immediate workaround (applied)

`T019 - Communication Templates` is being reworded to describe the notation without literal braces, so the test can run. This issue tracks the underlying harness gap.

---
*Found by the MJ Explorer Computer Use regression suite, run `run-20260727T202606Z` (2026-07-27).*

Contributor guide

Open the contributing guide

Research direction

Start with packages/AI/MJComputerUse/src/test-driver/ComputerUseTestDriver.ts around the CU-F7 guard, then read packages/AI/MJComputerUse/src/utils/variable-substitution.ts and its WHOLE_STRING and EMBEDDED matchers. Run the Computer Use regression covering T019 - Communication Templates. Done means literal mustache syntax can be represented without triggering the unresolved-variable guard, while genuine unresolved variables still fail fast.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.