getsentry / getsentry/XcodeBuildMCP

[Bug]: Swift Testing JSONL counts assertion issues as failed tests

Open
#533 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
6.4k
Forks
319
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

XcodeBuildMCP 2.7.0 JSONL test progress and the terminal build summary count Swift Testing assertion issues as failed tests. A single failed test can contain several failed `#expect` assertions, so this produces `failed > completed`, invents extra tests in the summary, and can report zero passes despite emitting passing test-case results.

Confirmed by replaying the original Xcode output through the installed, unmodified upstream parser/run-state modules, and independently with the small parser reproduction below. The reproduction uses no OptionsWheelTracker harness, project, simulator, or LLM. Installed source-map contents match the v2.7.0 sources. The relevant files are unchanged on upstream main at `e6ef59b49b44012c824f0a0de261c96142e37390`.

### Debug Output

Output from `xcodebuildmcp doctor doctor --output json`:

```json
{
"schema": "xcodebuildmcp.output.doctor-report",
"schemaVersion": "2",
"didError": false,
"error": null,
"data": {
"serverVersion": "2.7.0",
"checks": [
{
"name": "xcode",
"status": "ok",
"message": "Xcode 26.6 - Build version 17F113 (/Applications/Xcode.app/Contents/Developer)"
},
{
"name": "process-tree",
"status": "ok",
"message": "Running under Xcode: No; 4 process entries"
},
{
"name": "axe",
"status": "ok",
"message": "Available: Yes; UI automation: Yes; Video capture: Yes"
},
{
"name": "xcodemake",
"status": "ok",
"message": "Enabled: No; Binary: No; Makefile: Not checked"
},
{
"name": "mise",
"status": "warning",
"message": "Running under mise: No; Available: No"
},
{
"name": "debugger-dap",
"status": "ok",
"message": "Selected backend: lldb-cli; lldb-dap available: Yes"
},
{
"name": "manifest-tools",
"status": "ok",
"message": "Total tools: 82; Workflows: 15"
},
{
"name": "runtime-registration",
"status": "warning",
"message": "Runtime registry unavailable."
},
{
"name": "xcode-ide-bridge",
"status": "ok",
"message": "Workflow enabled: No; Connected: No; Proxied tools: 0"
},
{
"name": "sentry",
"status": "ok",
"message": "Enabled: Yes"
}
]
}
}
```

### Editor/Client

Codex desktop invoking the standalone Homebrew CLI. The isolated reproduction runs directly in Node.js.

### MCP Server Version

2.7.0, Homebrew installation. This is also the latest published release at the time of this report.

### LLM

Codex / GPT-6 during investigation. No model is involved in the reproduction.

### Steps to Reproduce

Save the following as `reproduce.mjs`. It connects the shipped event parser to the shipped run-state accumulator, as the production pipeline does. The input contains two ordinary, non-parameterized tests: one passes; one fails with three assertion issues.

```js
import { pathToFileURL } from 'node:url';

const moduleRoot = process.argv[2];
const { createXcodebuildEventParser } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-event-parser.js`)
);
const { createXcodebuildRunState } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-run-state.js`)
);
const events = [];
const state = createXcodebuildRunState({
operation: 'TEST',
onEvent: event => events.push(event),
});
const parser = createXcodebuildEventParser({
operation: 'TEST',
onEvent: event => state.push(event),
});
parser.onStdout([
'✔ Test "passes" passed after 0.001 seconds.',
'✘ Test "fails" recorded an issue at CountsTests.swift:10:3: Expectation failed: first',
'✘ Test "fails" recorded an issue at CountsTests.swift:11:3: Expectation failed: second',
'✘ Test "fails" recorded an issue at CountsTests.swift:12:3: Expectation failed: third',
'✘ Test "fails" failed after 0.001 seconds with 3 issues.',
'✘ Test run with 2 tests in 1 suite failed after 0.002 seconds with 3 issues.',
].join('\n') + '\n');
parser.flush();
state.finalize(false);
for (const { kind, fragment, ...data } of events) {
if (['test-case-result', 'test-progress', 'build-summary'].includes(fragment)) {
console.log(JSON.stringify({ event: `${kind}.${fragment}`, ...data }));
}
}

```

For a Homebrew installation, run:

```sh
node reproduce.mjs "$(brew --prefix xcodebuildmcp)/libexec/build"
```

For another installation, pass the package's `build` directory instead.

The original simulator run used `xcodebuildmcp simulator test ... --output jsonl` with `progress: true`. Its raw Swift Testing summary was:

```text
✘ Test run with 16 tests in 1 suite failed after 0.427 seconds with 22 issues.
```

The retained `.xcresult` summary independently reports 8 passed, 8 failed, 0 skipped. There are 16 named JSONL case results, also 8 passed and 8 failed. Some original tests were parameterized; these are named-test counts, not individual argument executions. The minimal reproduction avoids parameterization entirely.

### Expected Behavior

For the minimal reproduction, retain both case-result events and all three failure diagnostics, but report progress `completed: 2, failed: 1, skipped: 0` and terminal counts `totalTests: 2, passedTests: 1, failedTests: 1, skippedTests: 0`.

Assertion/diagnostic counts must remain distinct from test counts. Progress, case results, and the terminal summary should use a consistent counting unit.

### Actual Behavior

The installed 2.7.0 modules emit:

```jsonl
{"event":"test-result.test-case-result","operation":"TEST","test":"passes","status":"passed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":1,"failed":0,"skipped":0}
{"event":"test-result.test-case-result","operation":"TEST","test":"fails","status":"failed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":2,"failed":3,"skipped":0}
{"event":"test-result.build-summary","operation":"TEST","status":"FAILED","totalTests":3,"passedTests":0,"failedTests":3,"skippedTests":0}

```

In the original run, progress was `completed: 16, failed: 22`. The terminal summary was `totalTests: 25, passedTests: 0, failedTests: 25`. Replaying its raw output plus XcodeBuildMCP's own xcresult-failure enrichment reproduces that exact summary: 22 parsed issue diagnostics become 25 unique diagnostic fragments after enrichment.

The owning code paths are:

- [`parseSwiftTestingRunSummary`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/swift-testing-line-parsers.ts#L148) assigns the summary's `issues` count to `failed`.
- [`createXcodebuildEventParser`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-event-parser.ts#L253) suppresses native failed-result increments and then uses that summary value for failed progress.
- [`createTestSummaryFragment`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-run-state.ts#L98) takes `Math.max(state.failedTests, state.testFailures.length)`, conflating diagnostic fragments with failed tests again.

This report concerns the JSONL stream. The structured domain result has a separate xcresult-summary path; I am not claiming that every output mode reports these counts.

### Error Messages

The downstream runner correctly rejected the original contradictory stream with:

```text
test-result.build-summary: failed progress contradicts unique failed test-case results
```

That validation error is a consequence; the isolated upstream replay reproduces the wrong numbers before any downstream consumer runs.

Related history: #331 requested parser coverage including multiple expectations per test and was closed for inactivity; #374 documents aggregate parameterized-test entries; #389 changed progress reconciliation. I found no current issue reporting this concrete contradiction. Original downstream report: https://github.com/dpearson2699/ios-options-wheel-tracker/issues/1055.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/utils/swift-testing-line-parsers.ts, src/utils/xcodebuild-event-parser.ts, and src/utils/xcodebuild-run-state.ts, focusing on parseSwiftTestingRunSummary, createXcodebuildEventParser, and createTestSummaryFragment. Run the provided reproduce.mjs against the package build and verify that assertion diagnostics remain separate from test counts, with completed 2, failed 1, totalTests 2, passedTests 1, and failedTests 1.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
cli, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.