[Blocked] Benchmark/migration perf: parallelize compile_all (dominant cost) and the two benchmark legs

## Summary

The dashboard benchmark (and real bulk migrations) spend most of their wall-clock in the **`--compile` step**, which runs `kb-dashboard-cli` **once per dashboard, serially**. Profiling shows compile — not translation and not the co-occurrence probes — is the dominant cost, in *both* the Grafana and Datadog legs. There are two independent levers worth ~2–3× total speedup; both leave migration *output* unchanged.

Not urgent (filed post-investigation for July, after the June 29 release). Captured here with full data so it can be picked up cold.

## Baseline (G:500 D:671, against the local lab)

| Component | Time | % of total |
|---|---|---|
| **Total** | **2060.8s (34.3 min)** | 100% |
| Grafana leg | 1108.2s | 53.8% |
| Datadog leg | 952.5s | 46.2% |
| └ co-occurrence probes (grafana) | 43.2s | 2.1% |

- 1171 dashboards / 32,551 panels.
- Grafana per-dashboard translate: median **0.03s**, p95 0.59s, p99 2.01s, max 3.17s → translation is *negligible*.
- The two legs run **serially and additively** (1108 + 953 ≈ 2061).
- (This run included one-time downloads of uncached dashboards; compare cached-to-cached when measuring fixes.)

## Where the time goes (G:50 D:50 phase profile, timestamped stream)

Total 202.9s:

| Phase | Time | % of total |
|---|---|---|
| Datadog leg | 75.7s | 37% |
| **Grafana — compile** (`kb-dashboard-cli` / `uvx`, per-dashboard) | **68.8s** | **34%** |
| Grafana — migrate/translate (50 dashboards) | 57.6s | 28% |
| Grafana — schema discovery | 0.7s | <1% |
| └ of which co-occurrence probes | 5.0s | 2.5% |

Per-dashboard translate is tiny; the grafana leg is dominated by **compile**. The Datadog leg also runs `--compile`, so it is compile-heavy too.

## Root cause

`compile_all` compiles each YAML serially, one `kb-dashboard-cli` subprocess per dashboard:

```python
# observability_migration/targets/kibana/compile.py:48
def compile_all(yaml_dir, compiled_dir):
Path(compiled_dir).mkdir(parents=True, exist_ok=True)
results = []
for yaml_file in sorted(Path(yaml_dir).glob("*.yaml")): # <-- serial
out_dir = Path(compiled_dir) / yaml_file.stem
out_dir.mkdir(parents=True, exist_ok=True)
success, output = compile_yaml(yaml_file, out_dir) # <-- 1 subprocess each
results.append((yaml_file.name, success, output))
return results
```

For G:500 D:671 that's **1171 sequential subprocess starts** (each `kb-dashboard-cli compile --input-file `). It's embarrassingly parallel — each file is independent and results are collected by name.

## Proposed work (ranked)

**Lever A — parallelize `compile_all` (engine, `compile.py`).** Thread-pool the per-file compiles (bounded worker pool, e.g. `min(os.cpu_count(), 8)`), preserve deterministic result collection by name. Expected: compile 69s → ~10–15s on a multi-core box (~25–30% of total). **Low risk** (independent subprocesses; identical output), and it speeds up *real* migrations too, not just the benchmark. **Recommended first.**

**Lever B — run the two benchmark legs in parallel (harness, tools repo `server.py`).** The Grafana and Datadog legs are independent but serial. Running them concurrently → `max(1108, 953)` ≈ 1108s ≈ **~46% off**. Caveat: the legs currently share one `INPUT_DIR`/`OUTPUT_DIR`, so this needs per-leg dirs threaded through `_build_migrate_cmd` / `_parse_report`. Medium effort/risk; benchmark-only win.

**Lever C — parallelize per-dashboard migrate/translate (engine).** 28% of the grafana leg at G:50. Bigger engine change; lower priority.

Levers A and B compound but both consume cores, so combined < additive. On a core-starved CI box the parallel speedup is smaller.

## Secondary finding — wasted co-occurrence probes

Probes are only ~2% of wall-clock **on the local lab** (~6ms each), but the *count* is very wasteful: at G:500, **6766 probe queries, 6202 errored (91.7%), 1527 error-fallbacks** (batched probe 400s → per-candidate re-probe fan-out, issue #182/#188). On a **remote/Serverless target** each probe is network-bound (10s timeout on errors), so this could flip from negligible to dominant. Worth a follow-up: skip/avoid probing fields that will 400, and/or cache negative results more aggressively. Tracked-adjacent to #182/#188.

## Tooling already available (to support this work)

Instrumentation built during the investigation (on branches, not yet merged/pushed):

- **Engine** branch `perf/188-probe-instrumentation`:
- `SchemaResolver.probe_stats()` + a `[probe-stats] {json}` line gated on `OBS_MIGRATE_PROBE_STATS=1` (queries/seconds/errors/fallbacks/candidates).
- `scripts/bench_probe_ab.py` — interleaved A/B of `migrate --es-url` across two commits, with a request-layer probe counter that works on un-instrumented commits.
- **Tools** branch `feat/probe-stats-instrumentation` (elastic/observability-migration-platform-tools):
- Captures `probe_stats` per run into `benchmark_history.json` + renders it (Insights → Speed card, Duration tooltip).
- `OBS_BENCH_ES_URL` override so the benchmark can target a non-Cloud ES (e.g. the open local lab) creds-free.

## How to reproduce

```bash
# Local lab (open ES, security off):
bash scripts/full_local_demo.sh --recreate-lab # ES :19200, Kibana :15601, seeded metrics-*

# Headless phase profile (timestamped stream → compile vs migrate split):
OBS_MIGRATE_PROBE_STATS=1 obs-migrate migrate --source grafana --input-mode files \
--input-dir infra/grafana/dashboards --output-dir /tmp/out --compile \
--es-url http://localhost:19200 --esql-index 'metrics-*'

# Full benchmark via the harness (tools repo):
OBS_MIG_DIR= OBS_BENCH_ES_URL=http://localhost:19200 python3 server.py
# → Dashboards Benchmark, Schema discovery on
```

## Measurement plan for a fix

1. Headless G:50/D:50 with timestamped stream before/after → isolate the **compile** delta (baseline compile = 68.8s).
2. One cached G:500/D:671 end-to-end before/after for the total (baseline = 34.3 min, but re-baseline **cached** to remove download noise).

## Notes / caveats

- All numbers above are against the **local lab**, where probes are cheap (~6ms). The manager's benchmark historically ran against a remote/Serverless cluster, where probe latency would reshuffle the ranking (see secondary finding).
- These optimizations change *speed only*, not migration output (compile produces the same per-dashboard NDJSON; legs are independent).

Start in observability_migration/targets/kibana/compile.py at compile_all and inspect the tools repo server.py entry points _build_migrate_cmd and _parse_report. Run the headless G:50/D:50 profile, then compare cached before/after timings. Done means bounded parallel work preserves deterministic results and migration output, with benchmark legs using isolated directories if Lever B is attempted.

Written by the indexing model from the issue text.