github / github/github-mcp-server

Benchmark and improve tool-search ranking with indexed BM25

オープン
#2,996 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement request ai review
主要言語
Go
スター
33k
フォーク
5k
平均マージ
2日 1時間
マージ済み PR(30日)
52

説明

### Describe the feature or problem you’d like to solve

The GitHub MCP Server already exposes tool discovery/search functionality, but
there is no repeatable benchmark for measuring how reliably natural-language
queries retrieve the intended MCP tool.

As the tool inventory grows, a benchmark would make ranking changes measurable
and help prevent retrieval regressions.

This is separate from host-side deferred tool loading discussed in #1680. The
proposal only concerns ranking inside the server's existing tool-search
implementation.

### Proposed solution

Add a hand-labelled benchmark covering natural-language intents across the
server's major toolsets, then compare the current heuristic with an indexed
BM25 implementation.

A prototype benchmark contains 49 queries over 115 unique tools and produced:

| Strategy | Recall@1 | Recall@3 | MRR@10 | Query latency |
|---|---:|---:|---:|---:|
| Current heuristic | 71.4% | 81.6% | 0.792 | ~2.25 ms |
| Indexed BM25 | 71.4% | 87.8% | 0.802 | ~34 µs |
| Hybrid RRF | 73.5% | 87.8% | 0.823 | ~2.38 ms |

Indexed BM25 improved Recall@3 by 6.1 percentage points and was approximately
66x faster per query. The hybrid produced the strongest ranking quality.

Before submitting a PR, I would appreciate maintainer guidance on the preferred
scope:

1. Benchmark harness only
2. Benchmark plus indexed BM25
3. Benchmark plus a hybrid ranking experiment

### Example prompts or workflows

- "Find open issues assigned to me across repositories"
- "Read the files, reviews, and diff for a pull request"
- "Download logs for a failed workflow job"
- "Find exposed secrets detected in a repository"
- "Add an issue to a GitHub project"

### Additional context

The benchmark uses the complete current tool inventory and validates that every
labelled relevant tool exists. The prototype includes unit tests for indexing
tool names, descriptions, parameter names, and parameter descriptions.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start by locating the server’s existing tool-search implementation and reviewing the prototype benchmark and its unit tests for indexing tool names, descriptions, parameter names, and parameter descriptions. Confirm the preferred scope with maintainers, then measure the selected ranking strategy against the 49-query, 115-tool benchmark and report retrieval quality and latency without regressions.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
go
領域
backend-api-design, search
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。