getsentry / getsentry/XcodeBuildMCP

test_sim reports SUCCEEDED with 0 tests when per-call projectPath is ignored

Aberta
#517 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
TypeScript
Estrelas
6.4k
Forks
319
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

## Summary

`test_sim` appears to ignore a per-call `projectPath` and use the session default instead. When the two differ, it reported `SUCCEEDED` having run **zero tests**.

## What happened

I had several agents working concurrently in separate git worktrees of one repo. Session defaults were set once, pointing `projectPath` at the main checkout:

```
session_set_defaults
projectPath: /src/PinPoint.xcodeproj
scheme: PinPoint
simulatorName: iPhone 17 Pro
```

An agent working in `/.claude/worktrees/agent-XXXX/` called `test_sim` with its own worktree's `projectPath` and an `-only-testing` filter for a suite that exists **only** in that worktree. The call returned `SUCCEEDED` with 0 tests passed and 0 failed.

The suite it was asked to run did not exist in the project it actually built, so there was nothing to run and nothing to fail. The agent's only clue was the zero count; the status field said success.

## Why this is worth fixing above its size

A test tool that returns `SUCCEEDED` without running the requested tests is the worst available failure mode — it is indistinguishable from a pass at the point where a caller decides whether work is done. In an agent workflow nothing reads the count; it reads the status. Two of my agents would have reported tickets complete on the strength of it. Both worked around it by driving `xcodebuild` directly with their own `-derivedDataPath`, which is the right answer but only because they noticed.

## Suggested

Either honour a per-call `projectPath` over the session default, or fail loudly when one is passed and cannot be honoured. Failing that, a zero-test run should not report as `SUCCEEDED` — an `-only-testing` filter matching nothing is an error, not a pass.

## Environment

Multiple concurrent git worktrees of one project, one shared MCP session, iOS simulator destination. Repointing the shared session defaults was not an option, since a concurrent agent was relying on them.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Comece pela ferramenta test_sim e pelo tratamento de projectPath por chamada junto com session_set_defaults. Reproduza o relato com caminhos de worktree separados e um filtro -only-testing e, em seguida, verifique se o projeto solicitado é usado ou se uma execução filtrada para zero testes falha em vez de retornar SUCCEEDED.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
typescript
Domínio
devtools, mobile-dev, testing-qa
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Ativa
Clareza
Razoavelmente clara
Facilidade para iniciantes
52/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.