[Agentic Interface] Add OpenTelemetry O11Y to `example-mcp-dashbuilder`

Depends on #31.

After structured logging is in place, the server still lacks **distributed tracing** and **metrics**. There is no way to visualize tool call latency, correlate ES|QL query performance with Kibana export times, or connect dashbuilder traces with downstream Elasticsearch spans.

## Goals

1. Auto-instrument Node.js HTTP, ES client, and DNS via `@elastic/opentelemetry-node`.
2. Add manual spans for every MCP tool call with domain-specific attributes.
3. Bridge Pino logs into the OTel pipeline so logs, traces, and spans are correlated.
4. Provide a `withSpan` helper for instrumenting business logic without OTel API boilerplate.
5. All OTel functionality is **opt-in** — zero overhead when disabled.

## Design

### 1. OTel auto-instrumentation entry point

Create `server/src/utils/otel-init.ts`:

```typescript
/**
* Must be loaded BEFORE any other imports to hook into Node.js modules.
* Use via: node --import ./dist/utils/otel-init.js dist/index.js
*
* Environment:
* OTEL_ENABLED=true → opt-in to instrumentation (default: off)
* OTEL_SERVICE_NAME → service identity (default: mcp-dashbuilder)
* OTEL_EXPORTER_OTLP_ENDPOINT → collector (default: http://localhost:4318)
* OTEL_RESOURCE_ATTRIBUTES → e.g. deployment.environment=development
*/
if (process.env.OTEL_ENABLED === 'true') {
try {
require('@elastic/opentelemetry-node');
} catch (e) {
console.warn('[otel-init] @elastic/opentelemetry-node not loaded:', (e as Error).message);
}
}
export {};
```

Auto-instrumentation hooks the Elasticsearch JS client, `undici`/`fetch`, and DNS — giving ES|QL query spans for free.

### 2. Manual instrumentation helpers: `server/src/utils/instrumentation.ts`

```typescript
import { type Attributes, type Span, SpanStatusCode, trace } from '@opentelemetry/api';

const TRACER_NAME = 'mcp-dashbuilder';

export function getTracer() {
return trace.getTracer(TRACER_NAME);
}

export function withSpan(
name: string,
attributes: Attributes,
fn: (span: Span) => T,
): T {
return getTracer().startActiveSpan(name, { attributes }, (span) => {
try {
const result = fn(span);
if (result instanceof Promise) {
return result
.then((v) => { span.setStatus({ code: SpanStatusCode.OK }); return v; })
.catch((e) => {
span.recordException(e instanceof Error ? e : new Error(String(e)));
span.setStatus({ code: SpanStatusCode.ERROR });
throw e;
})
.finally(() => span.end()) as T;
}
span.setStatus({ code: SpanStatusCode.OK });
span.end();
return result;
} catch (e) {
span.recordException(e instanceof Error ? e : new Error(String(e)));
span.setStatus({ code: SpanStatusCode.ERROR });
span.end();
throw e;
}
});
}

/** Domain-specific span attribute keys */
export const SpanAttributes = {
MCP_TOOL: 'mcp.tool',
MCP_RESULT: 'mcp.result',
DASHBOARD_ID: 'dashboard.id',
CHART_TYPE: 'chart.type',
ESQL_QUERY: 'esql.query',
KIBANA_API_PATH: 'kibana.api.path',
KIBANA_API_STATUS: 'kibana.api.status',
USER_ID: 'user.id',
ERROR_TYPE: 'error.type',
} as const;
```

### 3. Instrument tool calls in `register-tool.ts`

Extend the existing timing/logging wrapper (from the logging spec) with a span:

```typescript
import { withSpan, SpanAttributes } from './instrumentation.js';

const wrappedHandler = async (args: Record) => {
return withSpan(`mcp_tool_${name}`, { [SpanAttributes.MCP_TOOL]: name }, async (span) => {
const parsed = schema.parse(args);
try {
const result = await handler(parsed);
span.setAttribute(SpanAttributes.MCP_RESULT, 'success');
logger.info({ tool: name, durationMs: /* ... */ }, 'tool.success');
return result;
} catch (err) {
span.setAttribute(SpanAttributes.MCP_RESULT, 'error');
logger.error({ tool: name, err }, 'tool.error');
throw err;
}
});
};
```

### 4. Instrument Kibana client

Add spans around Kibana API calls in `kibana-client.ts`:

```typescript
import { withSpan, SpanAttributes } from './instrumentation.js';

async function kibanaRequest(method: string, path: string, body?: unknown) {
return withSpan('kibana_api', {
[SpanAttributes.KIBANA_API_PATH]: path,
'http.method': method,
}, async (span) => {
const resp = await fetch(/* ... */);
span.setAttribute(SpanAttributes.KIBANA_API_STATUS, resp.status);
// ...
});
}
```

Elasticsearch client calls are auto-instrumented by `@elastic/opentelemetry-node` — no manual work needed.

### 5. Bridge Pino → OTel logs

Update the logger (from the logging spec) to include the `pino-opentelemetry-transport` target when OTel is active:

```typescript
// In server/src/utils/logger.ts
const targets: pino.TransportTargetOptions[] = [
{ target: 'pino-pretty', options: { destination: 2 }, level },
// ... file transport ...
];

if (process.env.OTEL_ENABLED === 'true') {
targets.push({ target: 'pino-opentelemetry-transport', options: {}, level });
}
```

This forwards structured log records to the OTel collector with automatic trace-id/span-id correlation.

### 6. Package scripts

Add OTel-enabled start scripts to `server/package.json`:

```json
{
"scripts": {
"dev:otel": "OTEL_ENABLED=true OTEL_SERVICE_NAME=mcp-dashbuilder node --import ./dist/utils/otel-init.js --import tsx src/index.ts",
"start:otel": "OTEL_ENABLED=true OTEL_SERVICE_NAME=mcp-dashbuilder node --import ./dist/utils/otel-init.js dist/index.js"
}
}
```

The default `dev` and `start` scripts remain unchanged — OTel is opt-in via the `:otel` variants.

## Environment Variables

| Variable | Default | Description |
|---|---|---|
| `OTEL_ENABLED` | `false` | Set `true` to activate instrumentation — OTel is **off by default** |
| `OTEL_SERVICE_NAME` | `mcp-dashbuilder` | Service identity in traces |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTel collector endpoint |
| `OTEL_RESOURCE_ATTRIBUTES` | _(unset)_ | Extra resource attrs, e.g. `deployment.environment=dev` |

## Dependencies

Add to `server/package.json`:

```
dependencies:
@opentelemetry/api: ^1.9.0
pino-opentelemetry-transport: ^3.0.0

optionalDependencies:
@elastic/opentelemetry-node: ^1.8.0
```

`@elastic/opentelemetry-node` is optional — if missing, auto-instrumentation is skipped but manual spans and log bridging still work via `@opentelemetry/api` (which is a no-op when no SDK is registered). Since `OTEL_ENABLED` defaults to off, installing the optional dep alone does nothing.

## Scope

### In scope
- `server/src/utils/otel-init.ts` — new file, auto-instrumentation loader
- `server/src/utils/instrumentation.ts` — new file, `withSpan` + `SpanAttributes`
- `server/src/utils/register-tool.ts` — wrap tool handlers in spans
- `server/src/utils/kibana-client.ts` — span around API calls
- `server/src/utils/logger.ts` — add `pino-opentelemetry-transport` target
- `server/package.json` — add deps, add `:otel` scripts

### Out of scope
- Metrics (histograms, counters) — follow-up
- Dashboard/APM setup in Kibana — documentation only
- HTTP Stream transport + trace context propagation from incoming requests (needed when auth lands)
- Preview app instrumentation

## Acceptance Criteria

- [ ] `npm run dev` works identically to today — no OTel overhead
- [ ] `npm run dev:otel` sends traces to a local OTel collector
- [ ] Every tool call produces a span named `mcp_tool_{name}` with `mcp.tool` and `mcp.result` attributes
- [ ] ES|QL queries appear as child spans (via auto-instrumentation of `@elastic/elasticsearch`)
- [ ] Kibana API calls appear as spans with path and status code
- [ ] Pino log records are forwarded to the OTel collector with trace correlation
- [ ] Without `OTEL_ENABLED=true`, zero OTel overhead — no spans, no log transport, no auto-instrumentation