Add OpenTelemetry support for production metrics and tracing

# Add OpenTelemetry Support for Production Metrics and Tracing

**Type:** Feature Request
**Priority:** High — production readiness blocker

---

## Summary

Add OpenTelemetry (OTEL) support to MemberJunction so that production deployments can export metrics and traces to any OTEL-compatible backend (Azure Monitor, AWS CloudWatch via OTEL Collector, Grafana, Datadog, etc.) for alerting and observability. The implementation must be **config-driven**, **zero-overhead when disabled**, and **fully backward compatible** — existing MJAPI installations that don't opt in must experience zero breakage.

## Motivation

MJ currently has an in-process `TelemetryManager` (in `@memberjunction/core`) that tracks RunView, RunQuery, AI, Engine, Cache, and Network operations with timing, pattern detection, and optimization insights. This is useful for development profiling but **cannot export data to external monitoring systems** for production alerting.

Production deployments need:
- **Latency alarms**: alert when RunView p95 > 2s, AI calls p99 > 30s, etc.
- **Error rate monitoring**: alert when error count spikes
- **Cache effectiveness**: track hit/miss ratios over time
- **AI cost tracking**: token usage by model over time
- **No vendor lock-in**: OTLP is the standard protocol accepted by all major monitoring platforms (Azure, AWS, GCP, Datadog, Grafana, etc.)

## Design

### Architecture Overview

```
┌─────────────────────────────────────────────────────────────────┐
│ @memberjunction/core (small addition) │
│ ┌──────────────────────────────────────────┐ │
│ │ TelemetryManager │ │
│ │ + OnEvent() callback hook │ ← NEW │
│ │ - StartEvent() / EndEvent() (existing) │ │
│ └──────────────┬───────────────────────────┘ │
│ │ events fire callback │
│ ▼ │
│ @memberjunction/server-telemetry (NEW package) │
│ ┌──────────────────────────────────────────┐ │
│ │ │ │
│ │ bootstrap.ts (ESM --import entrypoint) │ │
│ │ - Reads mj.config.cjs │ │
│ │ - Creates MeterProvider + Exporter │ │
│ │ - Creates TracerProvider (optional) │ │
│ │ - Stores in global object store │ │
│ │ - No-op if config missing/disabled │ │
│ │ │ │
│ │ OTELMiddleware extends │ │
│ │ BaseServerMiddleware │ │
│ │ - Hooks TelemetryManager.OnEvent() │ │
│ │ - Records OTEL metrics from events │ │
│ │ - HTTP request duration (pre/post) │ │
│ │ - Apollo plugin for GraphQL metrics │ │
│ │ - Graceful no-op if providers absent │ │
│ │ │ │
│ └──────────────────────────────────────────┘ │
│ │ │
│ ▼ OTLP (http/protobuf or gRPC) │
│ ┌──────────────────────────────────────────┐ │
│ │ Any OTEL Backend (zero MJ code needed) │ │
│ │ Azure Monitor │ AWS CloudWatch │ Grafana │ │
│ │ Datadog │ Jaeger │ OTEL Collector │ │
│ └──────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```

### Key Design Decisions

#### 1. Bridge TelemetryManager events → OTEL (not replace)

The existing `TelemetryManager` already instruments all MJ-meaningful code paths (RunView, AI, Engine, Cache, etc.). Rather than duplicating instrumentation, we add a simple event callback (`OnEvent`) to TelemetryManager and let the OTEL layer listen. This means:

- **Zero new instrumentation code** in MJCore, MJServer, or any existing package
- Existing in-process pattern detection and optimization insights remain untouched
- OTEL layer is a pure consumer of events that already exist

#### 2. Metrics first, traces optional

For production alerting, **metrics** (histograms, counters, gauges) are what drive alarms. Traces are useful for debugging but secondary. The implementation prioritizes metrics and makes traces opt-in.

Planned metrics:

| Metric Name | Type | Attributes | Source |
|---|---|---|---|
| `mj.runview.duration` | Histogram | `entity`, `cached`, `result_type` | RunView events |
| `mj.runquery.duration` | Histogram | `query_name` | RunQuery events |
| `mj.ai.duration` | Histogram | `model`, `operation_type` | AI events |
| `mj.ai.tokens` | Counter | `model`, `direction` (input/output) | AI events |
| `mj.engine.load.duration` | Histogram | `engine_class`, `operation` | Engine events |
| `mj.cache.operations` | Counter | `cache_type`, `status` (hit/miss/stale) | Cache events |
| `mj.http.request.duration` | Histogram | `method`, `route`, `status_code` | Pre/PostRoute middleware |
| `mj.graphql.operation.duration` | Histogram | `operation_name`, `operation_type` | Apollo plugin |
| `mj.errors` | Counter | `category`, `operation` | EndEvent with error |

#### 3. No OTEL auto-instrumentation in Phase 1

OTEL's auto-instrumentation for ESM requires `--experimental-loader=@opentelemetry/instrumentation/hook.mjs` — an extra unstable flag. Since TelemetryManager already captures all MJ-meaningful operations, auto-instrumentation provides marginal value for the alarm use case. It can be added later (Phase 2) when OTEL's ESM support stabilizes, for full SQL query spans and distributed trace propagation.

#### 4. Bootstrap lives in the published package, not MJAPI

The `bootstrap.ts` file lives inside `@memberjunction/server-telemetry` and is exposed as a subpath export. It's invoked via:

```bash
node --import @memberjunction/server-telemetry/bootstrap ...
```

MJAPI only needs to add one flag to its start script. No new files in MJAPI. The bootstrap file is published with the npm package.

#### 5. Full backward compatibility (CRITICAL)

MJAPI is not a published package — it's an app template users customize. The implementation **must not break** any existing MJAPI installation regardless of whether they adopt this feature.

Graceful degradation chain:

| Scenario | Behavior |
|---|---|
| Package installed + config + `--import` flag | Full metrics and traces exported via OTLP |
| Package installed + config, **no** `--import` flag | OTELMiddleware creates providers at middleware init time (slightly later, but fully functional). Metrics still work. |
| Package installed, **no** config section | Bootstrap and middleware both detect missing config → no-op, zero overhead |
| Package **not installed** | Nothing to discover via ClassFactory, nothing loaded, zero impact |
| Older MJAPI that never adds the flag | Falls into "no flag" scenario above — still works if package + config present, or is invisible if not |

**No scenario causes a crash, error, or degraded behavior for existing functionality.**

The OTELMiddleware's `Initialize()` method must detect whether bootstrap has run (by checking global object store for providers). If providers exist, use them. If not, attempt to create them from config. If no config, become a complete no-op. Every public method must guard against the "not initialized" state.

---

## Changes Required

### 1. `@memberjunction/core` — Small Addition

**File:** `packages/MJCore/src/generic/telemetryManager.ts`

Add an event callback mechanism to `TelemetryManager`:

```typescript
// Callback type
type TelemetryEventCallback = (
event: TelemetryEvent,
type: 'start' | 'end'
) => void;

// On TelemetryManager class:
private _eventCallbacks: TelemetryEventCallback[] = [];

/**
* Register a callback to be notified when telemetry events start or end.
* Callbacks are called synchronously. They MUST NOT throw — exceptions
* are swallowed to prevent observers from breaking the producer.
* Returns an unsubscribe function.
*/
public OnEvent(callback: TelemetryEventCallback): () => void {
this._eventCallbacks.push(callback);
return () => {
const idx = this._eventCallbacks.indexOf(callback);
if (idx >= 0) this._eventCallbacks.splice(idx, 1);
};
}

// Called from existing StartEvent() and EndEvent() methods:
private notifyCallbacks(event: TelemetryEvent, type: 'start' | 'end'): void {
for (const cb of this._eventCallbacks) {
try { cb(event, type); } catch { /* swallow — observers must not break producer */ }
}
}
```

This is fully backward compatible — no existing behavior changes, just a new optional hook. The unsubscribe function follows standard observable patterns.

### 2. `@memberjunction/server-telemetry` — New Package

**Location:** `packages/ServerTelemetry/`

#### `bootstrap.ts` — ESM `--import` Entrypoint

Exposed as subpath export: `@memberjunction/server-telemetry/bootstrap`

Responsibilities:
1. Read `mj.config.cjs` via cosmiconfig (same pattern as server-bootstrap)
2. Check `openTelemetry.enabled` — if false or missing, return immediately (no-op)
3. Check OTEL standard env vars (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`) — these override config file values
4. Create `MeterProvider` with OTLP exporter and `PeriodicExportingMetricReader`
5. Optionally create `TracerProvider` if `traces.enabled === true`
6. Store providers in `GetGlobalObjectStore()` under a well-known key
7. Register a graceful shutdown hook to flush pending metrics on SIGTERM

This runs before any application code, which is ideal for OTEL (but not required — see backward compat).

#### `OTELMiddleware.ts` — BaseServerMiddleware Implementation

```typescript
@RegisterClass(BaseServerMiddleware, 'OpenTelemetry')
export class OTELMiddleware extends BaseServerMiddleware {
get Label(): string { return 'OpenTelemetry'; }
get Enabled(): boolean { /* check config */ }
}
```

**`Initialize()`:**
1. Check global object store for providers created by bootstrap
2. If not found, attempt to create providers from config (handles "no --import flag" case)
3. If still no providers (no config), mark as disabled and return
4. Create OTEL instruments (histograms, counters) using `MeterProvider`
5. Register callback on `TelemetryManager.Instance.OnEvent()`
6. In the callback:
- On `'end'` events: record duration to appropriate histogram based on `event.category`
- Map `event.params` fields to OTEL attributes (entity name, model, cache status, etc.)
- For AI events: also record token counters
- For Cache events: increment hit/miss counters

**`GetPreAuthMiddleware()`:**
- Returns middleware that stamps `req.__otelStartTime = performance.now()` and extracts route info

**`GetPostRouteMiddleware()`:**
- Returns middleware that computes request duration and records to `mj.http.request.duration` histogram with `method`, `route`, `status_code` attributes

**`GetApolloPlugins()`:**
- Returns Apollo Server plugin that tracks GraphQL operation name, type (query/mutation/subscription), and duration

#### `package.json` Subpath Exports

```json
{
"name": "@memberjunction/server-telemetry",
"type": "module",
"exports": {
".": "./dist/index.js",
"./bootstrap": "./dist/bootstrap.js"
}
}
```

### 3. `@memberjunction/server` (MJServer) — Config Schema Addition

**File:** `packages/MJServer/src/config.ts`

Add `openTelemetry` section to the config schema:

```typescript
const openTelemetrySchema = z.object({
enabled: zodBooleanWithTransforms().default(false),
serviceName: z.string().default('mjapi'),
exporterEndpoint: z.string().optional(),
exporterProtocol: z.enum(['http/protobuf', 'http/json', 'grpc']).default('http/protobuf'),
metrics: z.object({
enabled: zodBooleanWithTransforms().default(true),
intervalMs: z.number().default(60000),
}).default({}),
traces: z.object({
enabled: zodBooleanWithTransforms().default(false),
sampleRate: z.number().min(0).max(1).default(1.0),
}).default({}),
}).optional();
```

Environment variable overrides (standard OTEL env vars always win when set):
- `OTEL_EXPORTER_OTLP_ENDPOINT` → overrides `exporterEndpoint`
- `OTEL_SERVICE_NAME` → overrides `serviceName`
- `OTEL_TRACES_SAMPLER_ARG` → overrides `sampleRate`

### 4. MJAPI — Start Script Update (Optional, Recommended)

Recommended update to `package.json` start script:

```diff
- "start": "node --experimental-specifier-resolution=node --import ./register.js -r dotenv/config ./src/index.ts"
+ "start": "node --experimental-specifier-resolution=node --import @memberjunction/server-telemetry/bootstrap --import ./register.js -r dotenv/config ./src/index.ts"
```

**Important:** If the `@memberjunction/server-telemetry` package is not installed, the `--import` will fail. Users should only add this flag when they install the package. This is consistent with how other optional integrations work. The system must work without this flag (the OTELMiddleware handles late initialization as a fallback).

---

## Configuration Example

```javascript
// mj.config.cjs
module.exports = {
// ... existing config ...

// Existing in-process telemetry (unchanged)
telemetry: {
enabled: true,
level: 'standard'
},

// NEW: OpenTelemetry export
openTelemetry: {
enabled: true,
serviceName: 'mjapi-prod',
exporterEndpoint: 'http://localhost:4318',
metrics: {
enabled: true,
intervalMs: 60000 // flush metrics every 60s
},
traces: {
enabled: false // opt-in when needed for debugging
}
}
};
```

---

## Alarm Examples (Post-Implementation)

Once metrics flow via OTLP, alarms are configured in any backend:

**Azure Monitor:**
```
Alert: mj.runview.duration p95 > 2000ms for 5 minutes
Alert: mj.errors count > 50 in 10 minutes
Alert: mj.ai.duration p99 > 30000ms
```

**AWS CloudWatch (via OTEL Collector):**
Same metrics, same thresholds, different UI.

**Grafana / Prometheus:**
```promql
histogram_quantile(0.95, rate(mj_runview_duration_bucket{entity="Users"}[5m])) > 2
```

All backends consume the same OTLP data. Zero vendor lock-in.

---

## Local Development

For local profiling, options include:
- **Existing TelemetryManager GraphQL API** — already works, no OTEL needed
- **OTEL ConsoleExporter** — set exporter to console for stdout JSON output
- **Jaeger all-in-one** — `docker run -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one` — accepts OTLP natively, web UI at localhost:16686 (Jaeger SDKs are retired since 2022, but the backend fully supports OTLP ingestion)
- **Aspire Dashboard** — Microsoft's lightweight OTEL dashboard, single container
- **Docker workbench** — could add optional Jaeger service to the existing workbench compose file

---

## NPM Dependencies (New Package)

```json
{
"dependencies": {
"@opentelemetry/api": "^1.x",
"@opentelemetry/sdk-node": "^0.x",
"@opentelemetry/sdk-metrics": "^1.x",
"@opentelemetry/sdk-trace-node": "^1.x",
"@opentelemetry/exporter-metrics-otlp-proto": "^0.x",
"@opentelemetry/exporter-trace-otlp-proto": "^0.x",
"@opentelemetry/resources": "^1.x",
"@opentelemetry/semantic-conventions": "^1.x"
},
"peerDependencies": {
"@memberjunction/core": "^2.x",
"@memberjunction/server": "^2.x",
"@memberjunction/global": "^2.x"
}
}
```

---

## Phase 2 (Future, Not In Scope)

When OTEL's ESM auto-instrumentation stabilizes (they're moving to `module.register()` which eliminates the `--experimental-loader` flag):

- Add auto-instrumentation for SQL query text in spans (`tedious`, `pg`)
- HTTP-level distributed trace context propagation (W3C Trace Context headers)
- Full request waterfall visualization across services
- Auto-instrumentation for `ioredis` (Redis pub/sub tracing)
- Consider bridging TelemetryManager insights as OTEL events/logs

---

## Testing

- **Unit tests:** Verify metrics are recorded with correct attributes when TelemetryManager events fire
- **Unit tests:** Verify no-op behavior when config is missing or disabled
- **Unit tests:** Verify graceful degradation when bootstrap hasn't run (no --import flag)
- **Unit tests:** Verify unsubscribe function works correctly
- **Integration test:** Start MJAPI with OTEL enabled, verify metrics appear at a test collector endpoint
- **Backward compat test:** Start MJAPI without the package installed, verify zero errors
- **Docker workbench:** Add optional Jaeger all-in-one service for local dev testing