[Epic] Add support for Kibana Discover-style search (DSL/KQL + runtime fields) in the Threat Hunt Workbench

# [Epic] Classic Discover-style search (DSL/KQL + runtime fields) in the Threat Hunt Workbench

> **Status:** Draft
> **Author:** David Elgut (PM, Security AI)
> **Date:** 2026-07-07
> **Epic Link:** *(to be filled when published as a GitHub issue)*
> **Target Release:** TBD

---

## Background

The [Elastic Security MCP App](https://github.com/elastic/example-mcp-app-security)'s Threat Hunt Workbench (`threat-hunt` tool, [threat-hunt.ts](https://github.com/elastic/example-mcp-app-security/blob/main/src/tools/threat-hunt.ts)) gives analysts an interactive query surface inside Claude — explore indices, inspect mappings, write and run queries, visualize results. Today that surface speaks exactly one query language: the `execute-esql` tool ([threat-hunt.ts:119-136](https://github.com/elastic/example-mcp-app-security/blob/main/src/tools/threat-hunt.ts#L119-L136)) sends every query to `POST _query` via `EsqlService`. There is no tool that executes a classic Discover-style search (DSL / `query_string` / KQL) against `_search`, and no support for runtime fields.

A customer surfaced this gap directly:

> "I have been using the Security MCP quite a bit lately. It seems it is limited to ES|QL which is a blocker when I want to use runtime fields. I am wondering if there any plan to enable use of Discover classic searching via MCP?"

This tracks with what a repo-wide search confirms: ES|QL's handling of runtime fields defined on a data view/index pattern is a known-but-informally-documented gap relative to classic Discover search — there's no single Kibana tracking issue to cite as root cause, but the behavioral difference is real and blocking for this workflow.

**On the evidence base:** this is currently a single, specific report rather than a volume signal. [Epic #37](https://github.com/elastic/example-mcp-app-security/issues/37) shipped `mcp_tool_called` telemetry (including `execute-esql` call volume and success rate), but that data lands only in the internal stack-telemetry view today — the BigQuery golden table it depends on (Epic #37, Work Stream 3) hasn't shipped, so `execute-esql` error/failure-rate data isn't yet queryable to size how often this gap is hit. Once WS3/WS4 of Epic #37 land, pulling `execute-esql` failure rates (and, ideally, tagging runtime-field-related failures) would let us validate demand before or during implementation rather than relying on one report.

This also advances the same adoption goal Epic #37 measures: MCP App usage lives or dies on whether the tools it exposes cover the workflows analysts actually need. A hard, repo-confirmed capability gap that pushes a user back to the Kibana UI is a direct drag on the "did the MCP App replace the workflow" outcome the wider Security AI / MCP App investment is trying to prove out.

---

## User Story / Problem Statement

**As a** security analyst threat hunting through the MCP App,
**I want to** run classic Discover-style searches (KQL/DSL, honoring runtime fields defined on my data view) from the same workbench I use for ES|QL,
**so that** I don't have to abandon my MCP-native hunting session and switch back to the Kibana Discover UI just because a query depends on a runtime field.

### Problem Details

- **Single query language today.** `execute-esql` ([threat-hunt.ts:119-136](https://github.com/elastic/example-mcp-app-security/blob/main/src/tools/threat-hunt.ts#L119-L136)) is the only ad-hoc query path in the app, backed solely by `EsqlService` → `POST _query`.
- **`validate-query` doesn't fill the gap.** It supports KQL/EQL/ES|QL ([detection-rules.ts:178-195](https://github.com/elastic/example-mcp-app-security/blob/main/src/tools/detection-rules.ts#L178-L195)), but only to validate a *rule's* query at authoring time — it doesn't execute a search or return hunting results.
- **No `_search`/DSL execution tool exists.** There is no equivalent of `execute-esql` that hits `_search` with `query_string`/KQL and returns hits.
- **Runtime fields are unsupported.** Nothing in `src/` references runtime fields; a hunter relying on a runtime field defined in their data view gets no result (or an error) from `execute-esql`, and has no fallback inside the MCP App.
- **Forces a context switch.** The customer's ask is explicit: when they hit this wall, they leave Claude and go back to classic Discover — defeating the reason they're using the MCP App in the first place.

---

## The Solution

Add a second query-execution tool alongside `execute-esql` — e.g. `execute-search` — that runs a classic `_search` query (KQL or raw DSL) against a chosen index pattern, resolving and honoring runtime fields from the associated data view/index pattern. The Threat Hunt Workbench UI gains a language toggle (ES|QL vs. classic search), mirroring the choice analysts already have in Kibana Discover.

### UX: Before

An analyst hunting through the MCP App hits a query that needs a runtime field. `execute-esql` can't resolve it. They leave Claude, reopen Kibana Discover, and re-run the investigation there — losing the MCP-native session, entity graph, and conversational context.

### UX: After

- The analyst runs the same query as a classic search from within the Threat Hunt Workbench.
- Runtime fields resolve correctly because the tool queries through the data view rather than raw `_query`.
- The hunting session — and any entity investigation already in progress — stays inside Claude.

---

## Goals & Non-Goals

### Goals

- Ship an `execute-search` (or similarly named) tool that runs classic DSL/KQL queries via `_search`.
- Support runtime fields by resolving queries against the relevant data view/index pattern rather than a raw index string.
- Surface the new query mode in the Threat Hunt Workbench UI alongside the existing ES|QL editor.
- Preserve today's `execute-esql` behavior unchanged — this is additive, not a replacement.

### Non-Goals

- **Not replacing ES|QL.** ES|QL remains the default/primary query surface; classic search is an escape hatch for cases ES|QL can't cover.
- **Not a full Discover UI clone.** No saved searches, no histograms, no column customization in v1 — just query + hits.
- **Not solving the upstream ES|QL/runtime-field limitation itself.** This epic works around the gap at the MCP App layer; it does not change ES|QL's engine-level behavior.

---

## Work Streams

### Work Stream 1: Tool contract & backend

- [ ] Define the `execute-search` tool's input schema: query (KQL string or raw DSL JSON), target data view/index pattern, time range, size, sort
- [ ] Add a `SearchService`/`SearchClient` under `src/elastic/` that resolves the data view (for runtime field definitions) and issues `POST /_search`
- [ ] Decide KQL→DSL translation approach (reuse Kibana's `@kbn/es-query` if available, or accept raw DSL only for v1)
- [ ] Handle and surface runtime field resolution errors distinctly from generic query errors

### Work Stream 2: MCP tool registration

- [ ] Register `execute-search` via `registerTrackedAppTool` in [threat-hunt.ts](https://github.com/elastic/example-mcp-app-security/blob/main/src/tools/threat-hunt.ts), following the `execute-esql` pattern
- [ ] Update `manifest.json` with a description that helps the model choose correctly between `execute-esql` and `execute-search` (e.g., "use this when a query depends on a runtime field or when ES|QL can't express the query")

### Work Stream 3: Workbench UI

- [ ] Add a query-language toggle to the Threat Hunt Workbench view
- [ ] Render classic search hits (distinct from the ES|QL results table)

### Work Stream 4: Validation

- [ ] `npm run typecheck`
- [ ] Unit tests for the new service/client and tool handler
- [ ] Manual verification against the customer's original runtime-field scenario

---

## Success Criteria

### Metrics

| Metric | Current | Target |
|---|---|---|
| Query languages supported in Threat Hunt Workbench | 1 (ES|QL only) | 2 (ES|QL + classic search) |
| Runtime field support | None | Runtime fields resolve correctly via data view |
| `execute-esql` failure rate attributable to runtime-field queries | Unknown — not yet queryable (blocked on Epic #37 WS3 golden table) | Baseline once queryable, then track reduction post-launch |
| Reported "had to leave MCP App for Discover" feedback | At least 1 (this request) | 0 for runtime-field-blocked queries |

### Acceptance Checklist

- [ ] `execute-search` tool ships, registered and tracked like `execute-esql`
- [ ] A query against a runtime field defined on a data view returns correct results through the new tool
- [ ] Existing `execute-esql` behavior is unaffected
- [ ] Workbench UI lets the analyst pick a query mode

---

## Related Issues

- No duplicate or existing tracking issue found in `elastic/example-mcp-app-security`, `elastic/kibana`, `elastic/platform-initiatives`, or `elastic/security-team` — confirmed via search prior to filing.
- [elastic/example-mcp-app-security#22](https://github.com/elastic/example-mcp-app-security/issues/22) — multi-cluster support (unrelated, but same repo area).

---

## Stakeholders

| Role | Name | Responsibility |
|---|---|---|
| Product Manager | David Elgut | Requirements, prioritization, acceptance |
| Customer / Idea origin | *Unnamed — Slack feedback* | Use case, requirements input |
| Engineering (MCP App) | *TBD* | Technical feasibility, implementation |

Start with src/tools/threat-hunt.ts, especially the execute-esql registration, then review src/elastic/ and manifest.json for the existing tool structure. Run npm run typecheck and inspect the planned service, handler, UI toggle, and unit-test requirements before implementation. Done means classic search is registered, runtime-field queries return correct results, the workbench offers both modes, and execute-esql remains unchanged.

Written by the indexing model from the issue text.