[CNCF LFX Proposal] Jaeger Benchmarking the AI Assistant's MCP Tools and Skills

### CNCF Project

Jaeger

### Term

2026 Term 3 (Sep-Nov)

### Program Name

Benchmarking the AI Assistant's MCP Tools and Skills

### Program Description

## Description

Jaeger's AI assistant analyzes distributed traces through two layers: an **MCP server** exposing tools that fetch trace data (span details, critical paths, service topology), and a **Skills framework** — declarative Markdown files that carry domain expertise and narrate when and how to use those tools.

Both layers were designed by intuition. We have no empirical evidence for how tool shape and Skill wording change an LLM's reasoning trajectory. This program builds a repeatable evaluation harness so those choices can be A/B tested against deterministic fault scenarios and scored on trajectory metrics rather than prose quality:

- **Call error rate** — did the model produce valid tool parameters?
- **Steps to evidence** — how many calls before the root cause was in hand?
- **Context bloat** — how much raw JSON did the tools push into the window?
- **Root-cause accuracy** — did it land on the seeded fault, without hallucinating?

The scenarios are deliberately *trace-solvable*: faults whose cause is fully deducible from spans and span-derived metrics (Jaeger's SPM API), requiring no application logs. Isolating that class of incident is itself part of the research.

Prior art to draw on: the OpenTelemetry Demo's feature flags (`cartFailure`, `productCatalogFailure`, …) for reproducible incidents; RCA benchmarks such as RCA100, Cloud-OpsBench, OpenRCA and RCAEval for methodology; trajectory-level tool-use evaluations that score intermediate steps rather than final text; and open-source LLM evaluation platforms (Opik, Arize Phoenix, Langfuse) to run and score the loops locally.

The technical architecture is the mentee's to propose. Existing work: jaegertracing/jaeger#8440 (Skills framework) and the Jaeger MCP server.

## Expected outcomes

1. **Trace-solvable benchmark suite** — 5-10 documented, deterministic fault scenarios with known ground-truth root causes, each verified to be solvable from traces and span-derived metrics alone.
2. **Evaluation harness** — a local testing loop, built on an open-source evaluation framework, that programmatically runs an LLM agent against the suite through the Jaeger MCP server and Skills framework, capturing full trajectories.
3. **A/B variants of both layers**, implemented and measured: granular data-fetching tools vs. high-level analytical tools; strict step-by-step Skill narratives vs. loose goal-oriented ones.
4. **Metrics report** comparing trajectory metrics across tool + Skill combinations, resolving to a data-backed recommendation for the default Jaeger AI assistant configuration.
5. **Upstreamed changes** to the Jaeger MCP server and Skills that follow from that recommendation, with documentation so the harness can be re-run as both layers evolve.

### Technologies

Go, Python, TypeScript, OpenTelemetry, Model Context Protocol (MCP), LLMs / prompt engineering, Docker Compose

### Skills same as Technologies?

- [ ] Yes, the required skills are the same as the technologies listed above.

### Required/Desirable Skills

Go, Python or TypeScript, distributed tracing concepts, LLM tool use / function calling, prompt engineering, agentic evaluation

### Mentors

Jonah Kowall | @jkowall | jkowall@kowall.net | jkowall(?)
Yuri Shkuro | @yurishkuro | github@ysh.us | yurishkuro

### Upstream Issue URL

https://github.com/jaegertracing/jaeger/issues/9135

### Application Prerequisites

- [x] Resume
- [ ] Cover Letter
- [ ] School Enrollment Verification
- [ ] Participation Permission from school or employer
- [ ] Coding Challenge
- [x] Custom Prerequisite (fill in details below)

### Coding Challenge URL

_No response_

### Custom Prerequisite Name

Project Proposal

### Custom Prerequisite Description

Upload a proposal (guidelines: jaegertracing.io/mentorship/for-mentees) covering: (1) how you would isolate and verify "trace-solvable" incidents from general system failures; (2) your plan for the evaluation loop — which open-source framework and LLM, and how you would capture and score trajectory metrics; (3) concrete variations you would test, e.g. how a tool's JSON schema might change and how the matching .md Skill file would support it; (4) a timeline, with a research phase before coding.

### Custom Prerequisite — File Upload

- [x] Yes — completion of this task requires the mentee to submit a file.

---
**LFX program:** [CNCF - Jaeger: Benchmarking the AI Assistant's MCP Tools and Skills (2026 Term 3)](https://mentorship.lfx.linuxfoundation.org/project/72e84a43-6be8-4967-a077-53e06b769096)

Start by reading Jaeger issues #8440 and #9135, then inspect the existing MCP server and Skills framework. Define trace-solvable fault scenarios and propose the evaluation harness, trajectory metrics, tool and Skill variants, and research timeline; done means a documented benchmark suite, reproducible local evaluation loop, comparison report, and justified upstream changes.

Written by the indexing model from the issue text.