Feature request(MCP Analytics): Tool evals

Feature request

Is your feature request related to a problem?

While developing MCP tools, it's difficult to know whether agents will use them correctly before shipping.

Individual tools can be tested like normal code, but that doesn't answer questions like:

• Will the model choose the right tool for a given task?
• Will it call tools in the expected order?
• Will it generate the right arguments?
• Will the overall journey successfully complete the task?
• Will the behavior stay consistent across multiple runs?

Today, these issues are mostly discovered through manual testing or after the MCP server is already in use.

Describe the solution you'd like

Add MCP Evals that let developers define a scenario, run it multiple times, and inspect how the model uses their tools.

For example:

Scenario:
"Find customer John and show their latest order."

Run it 5 times and see:

• Which tools were called
• In what order
• Whether the expected tools were used
• Whether the task succeeded

The basic workflow would be:

define scenario → run eval → inspect results → tweak tools → run again

Describe alternatives you've considered

Manually testing the same scenarios through an MCP client, but this is harder to repeat consistently and compare across runs.

Additional context

Customer described this as "tests in CI, except for MCP tools."

https://posthog.slack.com/archives/C0B3E1Y576X/p1786642627861069

Debug info

- [ ] PostHog Cloud, region and project ID: [please provide, you can find both at https://app.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

The issue names no files, tests, or entry points. Start by locating the existing MCP tooling and analytics integrations, then define how scenarios, repeated runs, tool order and arguments, and task success will be represented; done means developers can run an eval and inspect comparable results across runs.