getsentry / getsentry/XcodeBuildMCP
[Bug]: Swift Testing JSONL counts assertion issues as failed tests
- 主要語言
- TypeScript
- 星號
- 6.4k
- 分支
- 319
- PR 合併指標
- 30 天內沒有已合併 PR
描述
### Bug Description
XcodeBuildMCP 2.7.0 JSONL test progress and the terminal build summary count Swift Testing assertion issues as failed tests. A single failed test can contain several failed `#expect` assertions, so this produces `failed > completed`, invents extra tests in the summary, and can report zero passes despite emitting passing test-case results.
Confirmed by replaying the original Xcode output through the installed, unmodified upstream parser/run-state modules, and independently with the small parser reproduction below. The reproduction uses no OptionsWheelTracker harness, project, simulator, or LLM. Installed source-map contents match the v2.7.0 sources. The relevant files are unchanged on upstream main at `e6ef59b49b44012c824f0a0de261c96142e37390`.
### Debug Output
Output from `xcodebuildmcp doctor doctor --output json`:
```json
{
"schema": "xcodebuildmcp.output.doctor-report",
"schemaVersion": "2",
"didError": false,
"error": null,
"data": {
"serverVersion": "2.7.0",
"checks": [
{
"name": "xcode",
"status": "ok",
"message": "Xcode 26.6 - Build version 17F113 (/Applications/Xcode.app/Contents/Developer)"
},
{
"name": "process-tree",
"status": "ok",
"message": "Running under Xcode: No; 4 process entries"
},
{
"name": "axe",
"status": "ok",
"message": "Available: Yes; UI automation: Yes; Video capture: Yes"
},
{
"name": "xcodemake",
"status": "ok",
"message": "Enabled: No; Binary: No; Makefile: Not checked"
},
{
"name": "mise",
"status": "warning",
"message": "Running under mise: No; Available: No"
},
{
"name": "debugger-dap",
"status": "ok",
"message": "Selected backend: lldb-cli; lldb-dap available: Yes"
},
{
"name": "manifest-tools",
"status": "ok",
"message": "Total tools: 82; Workflows: 15"
},
{
"name": "runtime-registration",
"status": "warning",
"message": "Runtime registry unavailable."
},
{
"name": "xcode-ide-bridge",
"status": "ok",
"message": "Workflow enabled: No; Connected: No; Proxied tools: 0"
},
{
"name": "sentry",
"status": "ok",
"message": "Enabled: Yes"
}
]
}
}
```
### Editor/Client
Codex desktop invoking the standalone Homebrew CLI. The isolated reproduction runs directly in Node.js.
### MCP Server Version
2.7.0, Homebrew installation. This is also the latest published release at the time of this report.
### LLM
Codex / GPT-6 during investigation. No model is involved in the reproduction.
### Steps to Reproduce
Save the following as `reproduce.mjs`. It connects the shipped event parser to the shipped run-state accumulator, as the production pipeline does. The input contains two ordinary, non-parameterized tests: one passes; one fails with three assertion issues.
```js
import { pathToFileURL } from 'node:url';
const moduleRoot = process.argv[2];
const { createXcodebuildEventParser } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-event-parser.js`)
);
const { createXcodebuildRunState } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-run-state.js`)
);
const events = [];
const state = createXcodebuildRunState({
operation: 'TEST',
onEvent: event => events.push(event),
});
const parser = createXcodebuildEventParser({
operation: 'TEST',
onEvent: event => state.push(event),
});
parser.onStdout([
'✔ Test "passes" passed after 0.001 seconds.',
'✘ Test "fails" recorded an issue at CountsTests.swift:10:3: Expectation failed: first',
'✘ Test "fails" recorded an issue at CountsTests.swift:11:3: Expectation failed: second',
'✘ Test "fails" recorded an issue at CountsTests.swift:12:3: Expectation failed: third',
'✘ Test "fails" failed after 0.001 seconds with 3 issues.',
'✘ Test run with 2 tests in 1 suite failed after 0.002 seconds with 3 issues.',
].join('\n') + '\n');
parser.flush();
state.finalize(false);
for (const { kind, fragment, ...data } of events) {
if (['test-case-result', 'test-progress', 'build-summary'].includes(fragment)) {
console.log(JSON.stringify({ event: `${kind}.${fragment}`, ...data }));
}
}
```
For a Homebrew installation, run:
```sh
node reproduce.mjs "$(brew --prefix xcodebuildmcp)/libexec/build"
```
For another installation, pass the package's `build` directory instead.
The original simulator run used `xcodebuildmcp simulator test ... --output jsonl` with `progress: true`. Its raw Swift Testing summary was:
```text
✘ Test run with 16 tests in 1 suite failed after 0.427 seconds with 22 issues.
```
The retained `.xcresult` summary independently reports 8 passed, 8 failed, 0 skipped. There are 16 named JSONL case results, also 8 passed and 8 failed. Some original tests were parameterized; these are named-test counts, not individual argument executions. The minimal reproduction avoids parameterization entirely.
### Expected Behavior
For the minimal reproduction, retain both case-result events and all three failure diagnostics, but report progress `completed: 2, failed: 1, skipped: 0` and terminal counts `totalTests: 2, passedTests: 1, failedTests: 1, skippedTests: 0`.
Assertion/diagnostic counts must remain distinct from test counts. Progress, case results, and the terminal summary should use a consistent counting unit.
### Actual Behavior
The installed 2.7.0 modules emit:
```jsonl
{"event":"test-result.test-case-result","operation":"TEST","test":"passes","status":"passed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":1,"failed":0,"skipped":0}
{"event":"test-result.test-case-result","operation":"TEST","test":"fails","status":"failed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":2,"failed":3,"skipped":0}
{"event":"test-result.build-summary","operation":"TEST","status":"FAILED","totalTests":3,"passedTests":0,"failedTests":3,"skippedTests":0}
```
In the original run, progress was `completed: 16, failed: 22`. The terminal summary was `totalTests: 25, passedTests: 0, failedTests: 25`. Replaying its raw output plus XcodeBuildMCP's own xcresult-failure enrichment reproduces that exact summary: 22 parsed issue diagnostics become 25 unique diagnostic fragments after enrichment.
The owning code paths are:
- [`parseSwiftTestingRunSummary`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/swift-testing-line-parsers.ts#L148) assigns the summary's `issues` count to `failed`.
- [`createXcodebuildEventParser`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-event-parser.ts#L253) suppresses native failed-result increments and then uses that summary value for failed progress.
- [`createTestSummaryFragment`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-run-state.ts#L98) takes `Math.max(state.failedTests, state.testFailures.length)`, conflating diagnostic fragments with failed tests again.
This report concerns the JSONL stream. The structured domain result has a separate xcresult-summary path; I am not claiming that every output mode reports these counts.
### Error Messages
The downstream runner correctly rejected the original contradictory stream with:
```text
test-result.build-summary: failed progress contradicts unique failed test-case results
```
That validation error is a consequence; the isolated upstream replay reproduces the wrong numbers before any downstream consumer runs.
Related history: #331 requested parser coverage including multiple expectations per test and was closed for inactivity; #374 documents aggregate parameterized-test entries; #389 changed progress reconciliation. I found no current issue reporting this concrete contradiction. Original downstream report: https://github.com/dpearson2699/ios-options-wheel-tracker/issues/1055.
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
從 src/utils/swift-testing-line-parsers.ts、src/utils/xcodebuild-event-parser.ts 和 src/utils/xcodebuild-run-state.ts 開始,重點關注 parseSwiftTestingRunSummary、createXcodebuildEventParser 和 createTestSummaryFragment。針對 package build 執行提供的 reproduce.mjs,並驗證 assertion diagnostics 仍與測試計數分開,其中 completed 2、failed 1、totalTests 2、passedTests 1 和 failedTests 1。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- node.js, typescript
- 領域
- cli, testing
- Issue 類型
- 缺陷
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 活躍度
- 活躍
- 描述清晰度
- 描述清楚
- 新手友好度
- 76/100