game-ci / game-ci/cli

`customParameters` with a quoted value (`-testCategory "a;b"`, as the Unity manual shows) silently runs 0 tests since the CLI delegation

Open
#259 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
6
Avg merge
1h 51m
Merged PRs (30d)
167

Description

**Bug description**

The Unity Test Framework command-line reference — which game.ci's own `customParameters` docs link to — says a semicolon-separated list "should be formatted as a string enclosed in quotation marks", e.g. `-testCategory "firstCategory;secondCategory"`. Those quotes are shell quotes: typed into a terminal, the shell removes them and Unity receives `firstCategory;secondCategory`.

Through the CLI's Docker path there is no layer that does that removal, so the quotes are delivered to Unity literally. Unity does not strip quotes from argv, the category filter matches nothing, and the run finishes with **`Test run completed. Exiting with code 0 (Ok). No tests were executed.`** — a green job that ran zero tests.

This is a behaviour change relative to the pre-CLI actions. unity-test-runner ≤ v4.3 / v5.0.0-beta.1 built the `docker run` argv with `@actions/exec`, whose `argStringToArray` toggles quote state and drops the quote characters, so the container env was already `CUSTOM_PARAMETERS=-testCategory !IgnoreCI;!Integration …` and the same workflow ran the full suite. Before #257 the CLI path crashed outright on this input (the inner `"` broke the `--env NAME="…"` wrapper); #257 correctly made the value reach the container verbatim, which turned the loud failure into this silent one. Escaping was the right fix at that layer — the remaining gap is in how `test.sh` splits `CUSTOM_PARAMETERS`.

**How to reproduce**

```yaml
- uses: game-ci/unity-test-runner@v4.4.0 # → game-ci CLI v0.1.57
with:
customParameters: -testCategory "!IgnoreCI;!Integration" -testHelperScreenshotDirectory /github/workspace/artifacts/Screenshots
```

Unity's own argv dump in the log (6000.0.44f1, but identical on every version in the matrix):

```
COMMAND LINE ARGUMENTS:

-testCategory
"!IgnoreCI;!Integration"

Test run completed. Exiting with code 0 (Ok). No tests were executed.
```

- CLI v0.1.57, all five jobs green with `total="0"`: https://github.com/nowsprinting/UnityTestExamples/actions/runs/34415045587
- Same workflow on unity-test-runner v5.0.0-beta.1 (pre-CLI), 210 playmode + 1852 editmode tests executed: https://github.com/nowsprinting/UnityTestExamples/actions/runs/32160674823

Where it happens: `dist/platforms/ubuntu/steps/test.sh` expands `$CUSTOM_PARAMETERS` unquoted (v0.1.57 line 234) so it word-splits only — by design no `eval`, per the comment at lines 220-227. `build.sh` line 163 does the same, so `unity-builder` is affected the same way. `build.ps1` (line 114) splits on whitespace and has the same limitation.

**Expected behavior**

Either

1. `CUSTOM_PARAMETERS` is split shell-style — quote removal, no expansion, no execution — so a value copied from the Unity manual works. One eval-free way: let `xargs` do the parsing and only ever `printf` the tokens back:

```sh
mapfile -t customParameters < <(xargs printf '%s\n' <<< "$CUSTOM_PARAMETERS")

"${runTests[@]}" "${COVERAGE_FLAGS[@]}" "${customParameters[@]}"
```

`xargs` honours single/double quotes and backslashes but never runs a shell, so the injection concern that ruled out `eval` doesn't apply. (`build.sh` / `build.ps1` would want the same.)

or, if the split-on-whitespace contract is intentional,

2. document on `customParameters` that values must **not** be quoted (the Unity manual's quoting is for interactive shells only), and ideally warn when a `"` or `'` is present in `CUSTOM_PARAMETERS`, since the current outcome — a green run that executed nothing — is the worst failure mode for a CI tool.

**Additional details**

- Related: #257 (verbatim env values — prerequisite for this to be observable), game-ci/unity-test-runner#310 (CLI delegation).
- The workaround on the user side is simply to drop the quotes (`-testCategory !IgnoreCI;!Integration`); inside the container that is word-split into exactly the two argv entries Unity expects, and it also works on the pre-CLI actions.

Contributor guide

Open the contributing guide

Research direction

Start with dist/platforms/ubuntu/steps/test.sh around lines 220-234, then compare the equivalent splitting in build.sh line 163 and build.ps1 line 114. Reproduce the quoted customParameters case and verify that the chosen behavior is consistent across the scripts, either preserving shell-style argument intent or clearly documenting the existing contract. Done means the reproduced workflow no longer silently reports a successful zero-test run, or the limitation is explicitly surfaced.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, docker, powershell, typescript
Domain
cli, devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.