JetBrains / JetBrains/mcp-server-plugin
[MCP Server] Return a structured test summary (failedOnly), not a raw SM/TeamCity dump
- Dominant language
- Kotlin
- Stars
- 134
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
`execute_run_configuration` is the only built-in way for an agent to run an IDE JUnit configuration, but the tool is shaped like a generic process launcher, not a test tool.
On a completed module run (IntelliJ IDEA 2026.2, JUnit configuration `tests: fx-agent-workspace`):
- **4488** tests finished in ~11s
- **7** failures, **33** ignored
- raw SM / TeamCity log on disk: **~13.8 MB**
- MCP `output` snapshot is truncated at the **beginning** (the `suiteTree*` listing), so the inline payload is the least useful part of the run
- `fullOutputPath` has the complete dump, but the agent then has to `rg '##teamcity[testFailed'` to recover what a developer would glance at in the Test tool window
- `exitCode=255` is easy to misread as a timeout / transport failure (`System.exit(-1)` from the JUnit starter), while a green run is `0`
Agents do not need the passing-test stream or the SM protocol. They need the same thing a human looks at first: **counts + failed tests + messages / stacks**.
This also fights the MCP server's own guidance in `intellij-community/plugins/mcp-server`: heavy-output tools should cap / paginate and not return a huge unfiltered `String`.
## What we need
After a test run configuration finishes, return a **structured summary** in the tool result itself (keep `fullOutputPath` as an optional dump for the rare case someone wants stdout).
Minimum useful payload:
```json
{
"exitCode": 255,
"status": "FAILED",
"started": 4488,
"finished": 4488,
"failed": 7,
"ignored": 33,
"durationMs": 11000,
"fullOutputPath": "/path/to/ij_run_....log",
"failures": [
{
"className": "com.example.FooTest",
"methodName": "bar()",
"message": "expected X but was Y",
"stackTrace": "..."
}
]
}
```
A `failedOnly` / `includeOutput` flag (as in community plugins) would be enough. Passing tests should stay out of the MCP payload by default.
## Why this is not covered by IJPL-199602
[IJPL-199602](https://youtrack.jetbrains.com/issue/IJPL-199602) ([MCP Server] Run tests tool) is **Fixed** in 2026.1.1 and shipped launching via `execute_run_configuration` (`configurationName` or `filePath` + `line`) plus raw output / `fullOutputPath`.
The ticket also asked for *listening for results*. Today that is still the SM event dump, not the IDE test tree an agent can consume. Community plugins already grew a second API for this gap, e.g. `get_test_results(failedOnly: true)` in [jiayun/intellij-mcp#5](https://github.com/jiayun/intellij-mcp/issues/5).
## Related
- This repo: #40 (dedicated JUnit / TestNG run tool; still open)
- [IJPL-248621](https://youtrack.jetbrains.com/issue/IJPL-248621) — MCP reports exit code 0 even when tests failed
- [IJPL-251389](https://youtrack.jetbrains.com/issue/IJPL-251389) — `execute_run_configuration` reports a false launch failure
If this belongs on YouTrack instead of GitHub, please clone it to **IJPL / MCP Server** as a follow-up to IJPL-199602 (structured result / `failedOnly`). I could not file there from this environment without a JetBrains Account session.
## Environment
- IntelliJ IDEA 2026.2 (built-in MCP, `com.intellij.mcpServer`)
- Client: Cursor agent via IDEA MCP `execute_run_configuration`
- Repro: any large JUnit module configuration; wait for exit; inspect the truncated `output` vs `fullOutputPath`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the execute_run_configuration implementation in the MCP server plugin and review the heavy-output guidance mentioned for intellij-community/plugins/mcp-server. Trace how completed test-run output and fullOutputPath are currently returned, then identify the existing result information available from the SM/TeamCity stream. Done means the tool result exposes counts, status, duration, failures, and optional fullOutputPath without passing tests by default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin
- Domain
- devtools, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100