aws / aws/agent-toolkit-for-aws

aws-ai-ml templates: boolean placeholders wrapped in list brackets make False truthy (ACCEPT_EULA, EVALUATE_BASE)

Open Beginner friendly
#233 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.6k
Forks
306
Avg merge
1d 1h
Merged PRs (30d)
51

Description

### Describe the bug

Two boolean config placeholders in the `aws-ai-ml` code templates are wrapped in square brackets, which Python parses as a **list literal**, not a placeholder token:

| file | line | current |
|---|---|---|
| `aws-ai-ml/references/model-deployment/code_templates/deploy-oss-sagemaker.py` | 28 | `ACCEPT_EULA = [ACCEPT_EULA]` |
| `aws-ai-ml/references/model-evaluation/code_templates/llmaaj_evaluator.py` | 38 | `EVALUATE_BASE = [TRUE_OR_FALSE]` |

Each exists twice — under `skills/core-skills/` and `plugins/aws-core/skills/` (identical copies, parity-enforced by `lint:skill-parity`), so 4 locations total.

This fails in two distinct ways.

**1. As shipped, the templates do not run.** The bracket contents resolve as a bare name:

```
NameError: name 'ACCEPT_EULA' is not defined
```

**2. After substitution, the value is a list, and `[False]` is truthy.** The brackets survive the fill-in, so the result is a single-element list rather than a bool:

```
substitute True -> value=[True] type=list bool(v)=True
substitute False -> value=[False] type=list bool(v)=True <-- declining reads as accepting
```

This is the serious half. `ACCEPT_EULA` feeds `model_builder.accept_eula` (`deploy-oss-sagemaker.py:44`) — a **model licence acceptance flag that cannot express "no"**. Both `True` and `False` produce a truthy value. `EVALUATE_BASE` feeds `evaluate_base_model=` (`llmaaj_evaluator.py:51`), so opting out silently still runs the base-model evaluation, costing an extra evaluation run.

### Expected Behavior

`ACCEPT_EULA` and `EVALUATE_BASE` should be plain booleans, so that setting them to `False` actually evaluates as `False` — matching every sibling template in this repo:

- `finetuning/code_templates/sft.py:34` → `ACCEPT_EULA = False # Set to True to accept the base model's End-User License Agreement`
- `finetuning/code_templates/dpo.py:34` → same
- `finetuning/code_templates/rlvr.py:34` → same
- `model-evaluation/code_templates/custom_scorer_evaluator.py:50` → `EVALUATE_BASE = False`

These two files are the only outliers.

### Current Behavior

The templates raise `NameError` as shipped, and after substitution always produce a truthy list — so `False` is unrepresentable for both flags.

### Reproduction Steps

No AWS account or credentials needed; the defect is in the assignment itself.

```bash
git clone https://github.com/aws/agent-toolkit-for-aws.git
cd agent-toolkit-for-aws

# 1. Confirm the four occurrences
grep -rn "= \[ACCEPT_EULA\]\|= \[TRUE_OR_FALSE\]" --include="*.py" .

# 2. As shipped -> NameError
python3 -c 'exec("ACCEPT_EULA = [ACCEPT_EULA]", {})'

# 3. After substitution -> list, and False is truthy
python3 - <<'PY'
for tok in ("True", "False"):
ns = {}
exec(f"ACCEPT_EULA = [{tok}]", ns)
v = ns["ACCEPT_EULA"]
print(f"substitute {tok:5s} -> {v!r} type={type(v).__name__} bool={bool(v)}")
PY
```

Output of step 3:

```
substitute True -> [True] type=list bool=True
substitute False -> [False] type=list bool=True
```

A static check finds them too — `ruff check --select F821 .` reports `Undefined name ACCEPT_EULA` and `Undefined name TRUE_OR_FALSE`.

### Plugin Version

`main` @ `9766f27` (`chore(agents-pay): bump OpenClaw package to 1.0.6…`, #225)

### AI Assistant

N/A — the defect is in the shipped template source, reproducible with plain `python3`.

### AI Assistant Version

N/A

### OS

Reproduced on macOS 15 (Darwin 25.2.0) with CPython 3.12; not platform-dependent.

### Other information

**Suggested fix** — use plain booleans, matching the sibling templates:

```python
ACCEPT_EULA = True # Meta/Llama only — remove this line for non-Meta models
EVALUATE_BASE = False # set to True to also evaluate the base model for comparison
```

`BUILTIN_METRICS = [METRICS_LIST]` in the same file should be **left as-is** — there the brackets are a genuine list literal, so it is a different case despite the similar shape.

**Related, not fixed here:** `custom_scorer_evaluator.py:31` declares `EVALUATE_BASE = "[EVALUATE_BASE]"` — a *quoted string* placeholder for a boolean. That is valid Python, but any non-empty string is also truthy, so it may share the same failure mode depending on how the placeholder is substituted. Worth a look, but I left it alone since the intended substitution mechanics aren't clear to me from outside.

I also noticed `deploy-jumpstart-sagemaker.py:159` re-imports `Path`, shadowing the module-level import from line 11 (`ruff F811`). Harmless today; mentioning it only because it surfaced in the same sweep.

Note: `tools/validate.py` passes both before and after this change — it validates skill manifests and parity, not template Python, which is why this got through.

Contributor guide

Open the contributing guide

Research direction

Start with the four aws-ai-ml template copies named in the issue and compare their boolean assignments with the listed sibling templates. Run the grep and Python reproduction, then check ruff F821 and tools/validate.py. Done means both flags are plain booleans in all four locations, False remains false after substitution, and parity validation passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.