JetBrains / JetBrains/mcp-server-plugin

[MCP Server] Return a structured test summary (failedOnly), not a raw SM/TeamCity dump

Open
#57 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Kotlin
Stars
134
Forks
42
PR merge metrics
No merged PRs in 30d

Description

## Problem

`execute_run_configuration` is the only built-in way for an agent to run an IDE JUnit configuration, but the tool is shaped like a generic process launcher, not a test tool.

On a completed module run (IntelliJ IDEA 2026.2, JUnit configuration `tests: fx-agent-workspace`):

- **4488** tests finished in ~11s
- **7** failures, **33** ignored
- raw SM / TeamCity log on disk: **~13.8 MB**
- MCP `output` snapshot is truncated at the **beginning** (the `suiteTree*` listing), so the inline payload is the least useful part of the run
- `fullOutputPath` has the complete dump, but the agent then has to `rg '##teamcity[testFailed'` to recover what a developer would glance at in the Test tool window
- `exitCode=255` is easy to misread as a timeout / transport failure (`System.exit(-1)` from the JUnit starter), while a green run is `0`

Agents do not need the passing-test stream or the SM protocol. They need the same thing a human looks at first: **counts + failed tests + messages / stacks**.

This also fights the MCP server's own guidance in `intellij-community/plugins/mcp-server`: heavy-output tools should cap / paginate and not return a huge unfiltered `String`.

## What we need

After a test run configuration finishes, return a **structured summary** in the tool result itself (keep `fullOutputPath` as an optional dump for the rare case someone wants stdout).

Minimum useful payload:

```json
{
"exitCode": 255,
"status": "FAILED",
"started": 4488,
"finished": 4488,
"failed": 7,
"ignored": 33,
"durationMs": 11000,
"fullOutputPath": "/path/to/ij_run_....log",
"failures": [
{
"className": "com.example.FooTest",
"methodName": "bar()",
"message": "expected X but was Y",
"stackTrace": "..."
}
]
}
```

A `failedOnly` / `includeOutput` flag (as in community plugins) would be enough. Passing tests should stay out of the MCP payload by default.

## Why this is not covered by IJPL-199602

[IJPL-199602](https://youtrack.jetbrains.com/issue/IJPL-199602) ([MCP Server] Run tests tool) is **Fixed** in 2026.1.1 and shipped launching via `execute_run_configuration` (`configurationName` or `filePath` + `line`) plus raw output / `fullOutputPath`.

The ticket also asked for *listening for results*. Today that is still the SM event dump, not the IDE test tree an agent can consume. Community plugins already grew a second API for this gap, e.g. `get_test_results(failedOnly: true)` in [jiayun/intellij-mcp#5](https://github.com/jiayun/intellij-mcp/issues/5).

## Related

- This repo: #40 (dedicated JUnit / TestNG run tool; still open)
- [IJPL-248621](https://youtrack.jetbrains.com/issue/IJPL-248621) — MCP reports exit code 0 even when tests failed
- [IJPL-251389](https://youtrack.jetbrains.com/issue/IJPL-251389) — `execute_run_configuration` reports a false launch failure

If this belongs on YouTrack instead of GitHub, please clone it to **IJPL / MCP Server** as a follow-up to IJPL-199602 (structured result / `failedOnly`). I could not file there from this environment without a JetBrains Account session.

## Environment

- IntelliJ IDEA 2026.2 (built-in MCP, `com.intellij.mcpServer`)
- Client: Cursor agent via IDEA MCP `execute_run_configuration`
- Repro: any large JUnit module configuration; wait for exit; inspect the truncated `output` vs `fullOutputPath`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the execute_run_configuration implementation in the MCP server plugin and review the heavy-output guidance mentioned for intellij-community/plugins/mcp-server. Trace how completed test-run output and fullOutputPath are currently returned, then identify the existing result information available from the SM/TeamCity stream. Done means the tool result exposes counts, status, duration, failures, and optional fullOutputPath without passing tests by default.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
devtools, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.