getsentry / getsentry/XcodeBuildMCP

[Bug]: Swift Testing JSONL counts assertion issues as failed tests

未關閉
#533 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
TypeScript
星號
6.4k
分支
319
PR 合併指標
30 天內沒有已合併 PR

描述

### Bug Description

XcodeBuildMCP 2.7.0 JSONL test progress and the terminal build summary count Swift Testing assertion issues as failed tests. A single failed test can contain several failed `#expect` assertions, so this produces `failed > completed`, invents extra tests in the summary, and can report zero passes despite emitting passing test-case results.

Confirmed by replaying the original Xcode output through the installed, unmodified upstream parser/run-state modules, and independently with the small parser reproduction below. The reproduction uses no OptionsWheelTracker harness, project, simulator, or LLM. Installed source-map contents match the v2.7.0 sources. The relevant files are unchanged on upstream main at `e6ef59b49b44012c824f0a0de261c96142e37390`.

### Debug Output

Output from `xcodebuildmcp doctor doctor --output json`:

```json
{
"schema": "xcodebuildmcp.output.doctor-report",
"schemaVersion": "2",
"didError": false,
"error": null,
"data": {
"serverVersion": "2.7.0",
"checks": [
{
"name": "xcode",
"status": "ok",
"message": "Xcode 26.6 - Build version 17F113 (/Applications/Xcode.app/Contents/Developer)"
},
{
"name": "process-tree",
"status": "ok",
"message": "Running under Xcode: No; 4 process entries"
},
{
"name": "axe",
"status": "ok",
"message": "Available: Yes; UI automation: Yes; Video capture: Yes"
},
{
"name": "xcodemake",
"status": "ok",
"message": "Enabled: No; Binary: No; Makefile: Not checked"
},
{
"name": "mise",
"status": "warning",
"message": "Running under mise: No; Available: No"
},
{
"name": "debugger-dap",
"status": "ok",
"message": "Selected backend: lldb-cli; lldb-dap available: Yes"
},
{
"name": "manifest-tools",
"status": "ok",
"message": "Total tools: 82; Workflows: 15"
},
{
"name": "runtime-registration",
"status": "warning",
"message": "Runtime registry unavailable."
},
{
"name": "xcode-ide-bridge",
"status": "ok",
"message": "Workflow enabled: No; Connected: No; Proxied tools: 0"
},
{
"name": "sentry",
"status": "ok",
"message": "Enabled: Yes"
}
]
}
}
```

### Editor/Client

Codex desktop invoking the standalone Homebrew CLI. The isolated reproduction runs directly in Node.js.

### MCP Server Version

2.7.0, Homebrew installation. This is also the latest published release at the time of this report.

### LLM

Codex / GPT-6 during investigation. No model is involved in the reproduction.

### Steps to Reproduce

Save the following as `reproduce.mjs`. It connects the shipped event parser to the shipped run-state accumulator, as the production pipeline does. The input contains two ordinary, non-parameterized tests: one passes; one fails with three assertion issues.

```js
import { pathToFileURL } from 'node:url';

const moduleRoot = process.argv[2];
const { createXcodebuildEventParser } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-event-parser.js`)
);
const { createXcodebuildRunState } = await import(
pathToFileURL(`${moduleRoot}/utils/xcodebuild-run-state.js`)
);
const events = [];
const state = createXcodebuildRunState({
operation: 'TEST',
onEvent: event => events.push(event),
});
const parser = createXcodebuildEventParser({
operation: 'TEST',
onEvent: event => state.push(event),
});
parser.onStdout([
'✔ Test "passes" passed after 0.001 seconds.',
'✘ Test "fails" recorded an issue at CountsTests.swift:10:3: Expectation failed: first',
'✘ Test "fails" recorded an issue at CountsTests.swift:11:3: Expectation failed: second',
'✘ Test "fails" recorded an issue at CountsTests.swift:12:3: Expectation failed: third',
'✘ Test "fails" failed after 0.001 seconds with 3 issues.',
'✘ Test run with 2 tests in 1 suite failed after 0.002 seconds with 3 issues.',
].join('\n') + '\n');
parser.flush();
state.finalize(false);
for (const { kind, fragment, ...data } of events) {
if (['test-case-result', 'test-progress', 'build-summary'].includes(fragment)) {
console.log(JSON.stringify({ event: `${kind}.${fragment}`, ...data }));
}
}

```

For a Homebrew installation, run:

```sh
node reproduce.mjs "$(brew --prefix xcodebuildmcp)/libexec/build"
```

For another installation, pass the package's `build` directory instead.

The original simulator run used `xcodebuildmcp simulator test ... --output jsonl` with `progress: true`. Its raw Swift Testing summary was:

```text
✘ Test run with 16 tests in 1 suite failed after 0.427 seconds with 22 issues.
```

The retained `.xcresult` summary independently reports 8 passed, 8 failed, 0 skipped. There are 16 named JSONL case results, also 8 passed and 8 failed. Some original tests were parameterized; these are named-test counts, not individual argument executions. The minimal reproduction avoids parameterization entirely.

### Expected Behavior

For the minimal reproduction, retain both case-result events and all three failure diagnostics, but report progress `completed: 2, failed: 1, skipped: 0` and terminal counts `totalTests: 2, passedTests: 1, failedTests: 1, skippedTests: 0`.

Assertion/diagnostic counts must remain distinct from test counts. Progress, case results, and the terminal summary should use a consistent counting unit.

### Actual Behavior

The installed 2.7.0 modules emit:

```jsonl
{"event":"test-result.test-case-result","operation":"TEST","test":"passes","status":"passed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":1,"failed":0,"skipped":0}
{"event":"test-result.test-case-result","operation":"TEST","test":"fails","status":"failed","durationMs":1}
{"event":"test-result.test-progress","operation":"TEST","completed":2,"failed":3,"skipped":0}
{"event":"test-result.build-summary","operation":"TEST","status":"FAILED","totalTests":3,"passedTests":0,"failedTests":3,"skippedTests":0}

```

In the original run, progress was `completed: 16, failed: 22`. The terminal summary was `totalTests: 25, passedTests: 0, failedTests: 25`. Replaying its raw output plus XcodeBuildMCP's own xcresult-failure enrichment reproduces that exact summary: 22 parsed issue diagnostics become 25 unique diagnostic fragments after enrichment.

The owning code paths are:

- [`parseSwiftTestingRunSummary`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/swift-testing-line-parsers.ts#L148) assigns the summary's `issues` count to `failed`.
- [`createXcodebuildEventParser`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-event-parser.ts#L253) suppresses native failed-result increments and then uses that summary value for failed progress.
- [`createTestSummaryFragment`](https://github.com/getsentry/XcodeBuildMCP/blob/v2.7.0/src/utils/xcodebuild-run-state.ts#L98) takes `Math.max(state.failedTests, state.testFailures.length)`, conflating diagnostic fragments with failed tests again.

This report concerns the JSONL stream. The structured domain result has a separate xcresult-summary path; I am not claiming that every output mode reports these counts.

### Error Messages

The downstream runner correctly rejected the original contradictory stream with:

```text
test-result.build-summary: failed progress contradicts unique failed test-case results
```

That validation error is a consequence; the isolated upstream replay reproduces the wrong numbers before any downstream consumer runs.

Related history: #331 requested parser coverage including multiple expectations per test and was closed for inactivity; #374 documents aggregate parameterized-test entries; #389 changed progress reconciliation. I found no current issue reporting this concrete contradiction. Original downstream report: https://github.com/dpearson2699/ios-options-wheel-tracker/issues/1055.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

從 src/utils/swift-testing-line-parsers.ts、src/utils/xcodebuild-event-parser.ts 和 src/utils/xcodebuild-run-state.ts 開始,重點關注 parseSwiftTestingRunSummary、createXcodebuildEventParser 和 createTestSummaryFragment。針對 package build 執行提供的 reproduce.mjs,並驗證 assertion diagnostics 仍與測試計數分開,其中 completed 2、failed 1、totalTests 2、passedTests 1 和 failedTests 1。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
node.js, typescript
領域
cli, testing
Issue 類型
缺陷
難度
3/5
預估耗時
1-2 天
活躍度
活躍
描述清晰度
描述清楚
新手友好度
76/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。