aws / aws/agent-toolkit-for-aws
aws-ai-ml templates: boolean placeholders wrapped in list brackets make False truthy (ACCEPT_EULA, EVALUATE_BASE)
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 306
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 51
Description
### Describe the bug
Two boolean config placeholders in the `aws-ai-ml` code templates are wrapped in square brackets, which Python parses as a **list literal**, not a placeholder token:
| file | line | current |
|---|---|---|
| `aws-ai-ml/references/model-deployment/code_templates/deploy-oss-sagemaker.py` | 28 | `ACCEPT_EULA = [ACCEPT_EULA]` |
| `aws-ai-ml/references/model-evaluation/code_templates/llmaaj_evaluator.py` | 38 | `EVALUATE_BASE = [TRUE_OR_FALSE]` |
Each exists twice — under `skills/core-skills/` and `plugins/aws-core/skills/` (identical copies, parity-enforced by `lint:skill-parity`), so 4 locations total.
This fails in two distinct ways.
**1. As shipped, the templates do not run.** The bracket contents resolve as a bare name:
```
NameError: name 'ACCEPT_EULA' is not defined
```
**2. After substitution, the value is a list, and `[False]` is truthy.** The brackets survive the fill-in, so the result is a single-element list rather than a bool:
```
substitute True -> value=[True] type=list bool(v)=True
substitute False -> value=[False] type=list bool(v)=True <-- declining reads as accepting
```
This is the serious half. `ACCEPT_EULA` feeds `model_builder.accept_eula` (`deploy-oss-sagemaker.py:44`) — a **model licence acceptance flag that cannot express "no"**. Both `True` and `False` produce a truthy value. `EVALUATE_BASE` feeds `evaluate_base_model=` (`llmaaj_evaluator.py:51`), so opting out silently still runs the base-model evaluation, costing an extra evaluation run.
### Expected Behavior
`ACCEPT_EULA` and `EVALUATE_BASE` should be plain booleans, so that setting them to `False` actually evaluates as `False` — matching every sibling template in this repo:
- `finetuning/code_templates/sft.py:34` → `ACCEPT_EULA = False # Set to True to accept the base model's End-User License Agreement`
- `finetuning/code_templates/dpo.py:34` → same
- `finetuning/code_templates/rlvr.py:34` → same
- `model-evaluation/code_templates/custom_scorer_evaluator.py:50` → `EVALUATE_BASE = False`
These two files are the only outliers.
### Current Behavior
The templates raise `NameError` as shipped, and after substitution always produce a truthy list — so `False` is unrepresentable for both flags.
### Reproduction Steps
No AWS account or credentials needed; the defect is in the assignment itself.
```bash
git clone https://github.com/aws/agent-toolkit-for-aws.git
cd agent-toolkit-for-aws
# 1. Confirm the four occurrences
grep -rn "= \[ACCEPT_EULA\]\|= \[TRUE_OR_FALSE\]" --include="*.py" .
# 2. As shipped -> NameError
python3 -c 'exec("ACCEPT_EULA = [ACCEPT_EULA]", {})'
# 3. After substitution -> list, and False is truthy
python3 - <<'PY'
for tok in ("True", "False"):
ns = {}
exec(f"ACCEPT_EULA = [{tok}]", ns)
v = ns["ACCEPT_EULA"]
print(f"substitute {tok:5s} -> {v!r} type={type(v).__name__} bool={bool(v)}")
PY
```
Output of step 3:
```
substitute True -> [True] type=list bool=True
substitute False -> [False] type=list bool=True
```
A static check finds them too — `ruff check --select F821 .` reports `Undefined name ACCEPT_EULA` and `Undefined name TRUE_OR_FALSE`.
### Plugin Version
`main` @ `9766f27` (`chore(agents-pay): bump OpenClaw package to 1.0.6…`, #225)
### AI Assistant
N/A — the defect is in the shipped template source, reproducible with plain `python3`.
### AI Assistant Version
N/A
### OS
Reproduced on macOS 15 (Darwin 25.2.0) with CPython 3.12; not platform-dependent.
### Other information
**Suggested fix** — use plain booleans, matching the sibling templates:
```python
ACCEPT_EULA = True # Meta/Llama only — remove this line for non-Meta models
EVALUATE_BASE = False # set to True to also evaluate the base model for comparison
```
`BUILTIN_METRICS = [METRICS_LIST]` in the same file should be **left as-is** — there the brackets are a genuine list literal, so it is a different case despite the similar shape.
**Related, not fixed here:** `custom_scorer_evaluator.py:31` declares `EVALUATE_BASE = "[EVALUATE_BASE]"` — a *quoted string* placeholder for a boolean. That is valid Python, but any non-empty string is also truthy, so it may share the same failure mode depending on how the placeholder is substituted. Worth a look, but I left it alone since the intended substitution mechanics aren't clear to me from outside.
I also noticed `deploy-jumpstart-sagemaker.py:159` re-imports `Path`, shadowing the module-level import from line 11 (`ruff F811`). Harmless today; mentioning it only because it surfaced in the same sweep.
Note: `tools/validate.py` passes both before and after this change — it validates skill manifests and parity, not template Python, which is why this got through.
Contributor guide
Research direction
Start with the four aws-ai-ml template copies named in the issue and compare their boolean assignments with the listed sibling templates. Run the grep and Python reproduction, then check ruff F821 and tools/validate.py. Done means both flags are plain booleans in all four locations, False remains false after substitution, and parity validation passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100