getsentry / getsentry/XcodeBuildMCP
[Bug]: Swift Testing JSONL counts assertion issues as failed tests
- Ngôn ngữ chính
- TypeScript
- Star
- 6.4k
- Fork
- 319
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
### Bug Description
XcodeBuildMCP 2.7.0 JSONL test progress and the terminal build summary count Swift Testing assertion issues as failed tests. A single failed test can contain several failed `#expect` assertions, so this produces `failed > completed`, invents extra tests in the summary, and can report zero passes despite emitting passing test-case results.
Confirmed by replaying the original Xcode output through the installed, unmodified upstream parser/run-state modules, and independently with the small parser reproduction below. The reproduction uses no OptionsWheelTracker harness, project, simulator, or LLM. Installed source-map contents match the v2.7.0 sources. The relevant files are unchanged on upstream main at `e6ef59b49b44012c824f0a0de261c96142e37390`.
### Debug Output
Output from `xcodebuildmcp doctor doctor --output json`:
```json
{
"schema": "xcodebuildmcp.output.doctor-report",
"schemaVersion": "2",
"didError": false,
"error": null,
"data": {
"serverVersion": "2.7.0",
"checks": [
{
"name": "xcode",
"status": "ok",
"message": "Xcode 26.6 - Build version 17F113 (/Applications/Xcode.app/Contents/Developer)"
},
{
"name": "process-tree",
"status": "ok",
"message": "Running under Xcode: No; 4 process entries"
},
{
"name": "axe",
"status": "ok",
"message": "Available: Yes; UI automation: Yes; Video capture: Yes"
},
{
"name": "xcodemake",
"status": "ok",
"message": "Enabled: No; Binary: No; Makefile: Not checked"
},
{
"name": "mise",
"status": "warning",
"message": "Running under mise: No; Available: No"
},
{
"name": "debugger-dap",
"status": "ok",
"message": "Selected backend: lldb-cli; lldb-dap available: Yes"
},
{
"name": "manifest-tools",
"status": "ok",
"message": "Total tools: 82; Workflows: 15"
},
{
"name": "runtime-registration",
"status": "warning",
"message": "Runtime registry unavailable."
},
{
"name": "xcode-ide-bridge",
"status": "ok",
"message": "Workflow enabled: No; Connected: No; Proxied tools: 0"
},
{
"name": "sentry",
"status": "ok",
"message": "Enabled: Yes"
}
]
}
}
```
### Editor/Client
Codex desktop invoking the standalone Homebrew CLI. The isolated reproduction runs directly in Node.js.
### MCP Server Version
2.7.0, Homebrew installation. This is also the latest published release at the time of this report.
### LLM
Codex / GPT-6 during investigation. No model is involved in the reproduction.
### Steps to Reproduce
Save the following as `reproduce.mjs`. It connects the shipped event parser to the shipped run-state accumulator, as the production pipeline does. The input contains two ordinary, non-parameterized tests: one passes; one fails with three assertion issues.
```js
import { pathToFileURL } from 'node:url';
const moduleRoot = process.argv[2];
const { createXcodebuildEventParser } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-event-parser.js`)
);
const { createXcodebuildRunState } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-run-state.js`)
);
const events = [];
const state = createXcodebuildRunState({
operation: 'TEST',
onEvent: event => events.push(event),
});
const parser = createXcodebuildEventParser({
operation: 'TEST',
onEvent: event => state.push(event),
});
parser.onStdout([
'✔ Test "passes" passed after 0.001 seconds.',
'✘ Test "fails" recorded an issue at CountsTests.swift:10:3: Expectation failed: first',
'✘ Test "fails" recorded an issue at CountsTests.swift:11:3: Expectation failed: second',
'✘ Test "fails" recorded an issue at CountsTests.swift:12:3: Expectation failed: third',
'✘ Test "fails" failed after 0.001 seconds with 3 issues.',
'✘ Test run with 2 tests in 1 suite failed after 0.002 seconds with 3 issues.',
].join('\n') + '\n');
parser.flush();
state.finalize(false);
for (const { kind, fragment, ...data } of events) {
if (['test-case-result', 'test-progress', 'build-summary'].includes(fragment)) {
console.log(JSON.stringify({ event: `${kind}.${fragment}`, ...data }));
}
}
```
For a Homebrew installation, run:
```sh
node reproduce.mjs "$(brew --prefix xcodebuildmcp)/libexec/build"
```
For another installation, pass the package's `build` directory instead.
The original simulator run used `xcodebuildmcp simulator test ... --output jsonl` with `progress: true`. Its raw Swift Testing summary was:
```text
✘ Test run with 16 tests in 1 suite failed after 0.427 seconds with 22 issues.
```
The retained `.xcresult` summary independently reports 8 passed, 8 failed, 0 skipped. There are 16 named JSONL case results, also 8 passed and 8 failed. Some original tests were parameterized; these are named-test counts, not individual argument executions. The minimal reproduction avoids parameterization entirely.
### Expected Behavior
For the minimal reproduction, retain both case-result events and all three failure diagnostics, but report progress `completed: 2, failed: 1, skipped: 0` and terminal counts `totalTests: 2, passedTests: 1, failedTests: 1, skippedTests: 0`.
Assertion/diagnostic counts must remain distinct from test counts. Progress, case results, and the terminal summary should use a consistent counting unit.
### Actual Behavior
The installed 2.7.0 modules emit:
```jsonl
{"event":"test-result.test-case-result","operation":"TEST","test":"passes","status":"passed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":1,"failed":0,"skipped":0}
{"event":"test-result.test-case-result","operation":"TEST","test":"fails","status":"failed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":2,"failed":3,"skipped":0}
{"event":"test-result.build-summary","operation":"TEST","status":"FAILED","totalTests":3,"passedTests":0,"failedTests":3,"skippedTests":0}
```
In the original run, progress was `completed: 16, failed: 22`. The terminal summary was `totalTests: 25, passedTests: 0, failedTests: 25`. Replaying its raw output plus XcodeBuildMCP's own xcresult-failure enrichment reproduces that exact summary: 22 parsed issue diagnostics become 25 unique diagnostic fragments after enrichment.
The owning code paths are:
- [`parseSwiftTestingRunSummary`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/swift-testing-line-parsers.ts#L148) assigns the summary's `issues` count to `failed`.
- [`createXcodebuildEventParser`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-event-parser.ts#L253) suppresses native failed-result increments and then uses that summary value for failed progress.
- [`createTestSummaryFragment`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-run-state.ts#L98) takes `Math.max(state.failedTests, state.testFailures.length)`, conflating diagnostic fragments with failed tests again.
This report concerns the JSONL stream. The structured domain result has a separate xcresult-summary path; I am not claiming that every output mode reports these counts.
### Error Messages
The downstream runner correctly rejected the original contradictory stream with:
```text
test-result.build-summary: failed progress contradicts unique failed test-case results
```
That validation error is a consequence; the isolated upstream replay reproduces the wrong numbers before any downstream consumer runs.
Related history: #331 requested parser coverage including multiple expectations per test and was closed for inactivity; #374 documents aggregate parameterized-test entries; #389 changed progress reconciliation. I found no current issue reporting this concrete contradiction. Original downstream report: https://github.com/dpearson2699/ios-options-wheel-tracker/issues/1055.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Start with src/utils/swift-testing-line-parsers.ts, src/utils/xcodebuild-event-parser.ts, and src/utils/xcodebuild-run-state.ts, focusing on parseSwiftTestingRunSummary, createXcodebuildEventParser, and createTestSummaryFragment. Run the provided reproduce.mjs against the package build and verify that assertion diagnostics remain separate from test counts, with completed 2, failed 1, totalTests 2, passedTests 1, and failedTests 1.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- node.js, typescript
- Lĩnh vực
- cli, testing
- Loại issue
- Lỗi
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 76/100