Expose provider-safe operational telemetry for embedded runtimes

Summary

Expand Duroxide's provider-safe observability contract so embedding applications can monitor queue latency, locks, timers, retries, replay, dispatchers, and provider operations without querying provider implementation tables.

This request comes from PilotSwarm's OpenTelemetry work. Duroxide should remain telemetry-backend-neutral: expose stable runtime/management observations and continue allowing applications to choose their own exporter.

What exists today

Duroxide already provides two complementary surfaces:

1. Runtime metrics through the Rust metrics facade, including orchestration/activity outcomes and durations, dispatcher counts/durations, queue gauges, active orchestrations, sub-orchestration metrics, and provider operation/error metrics.
2. Provider-safe management/runtime snapshots:
   • SystemMetrics
   • QueueDepths
   • per-instance orchestration stats
   • MetricsSnapshot process-local counters

These are useful, but embedders cannot currently retrieve many of the rich facade observations through a stable API. The current public structs do not expose queue age, lock lifecycle, timer lateness, replay cost, provider operation distributions, or dispatcher utilization.

Requested contract

Add a stable provider/runtime observability snapshot API, or evolve the existing structs compatibly, to expose the following.

Provider-wide queue health

Per queue (orchestrator, worker, timer):

• eligible/unlocked depth;
• locked/in-flight depth where meaningful;
• oldest eligible item age;
• enqueue-to-lock/start latency distribution or aggregate buckets;
• due timer count and timer lateness distribution;
• poll count, empty-poll count, fetched-item count, and fetch errors.

Clarify whether each value is a current gauge, process-local monotonic counter, provider-wide cumulative counter, or interval delta.

Lock and lease health

By lock kind where applicable:

• acquisition attempts, successes, and failures;
• acquisition wait duration;
• renewal attempts/failures;
• expired/lost locks;
• reclaimed work;
• session lease claimed/renewed/idle-expired/reclaimed events or counters.

This overlaps with the quiescence/session-lease observations requested in #38; reuse that contract rather than defining a competing one.

Runtime and dispatcher health

• configured orchestration/activity concurrency;
• active/in-flight orchestration and activity counts;
• dispatcher processing/execution duration aggregates;
• work fetched and completed by dispatcher;
• shutdown/drain state and quiescence;
• retry attempts by operation and outcome;
• continue-as-new and sub-orchestration outcomes;
• poison and nondeterminism failures.

Replay and history

• replay duration;
• replayed event count;
• history event and byte distributions at completion;
• replay/nondeterminism failure classifications;
• acknowledgement/commit duration and conflicts.

Provider operations

Expose bounded operation-level aggregates for fetch, enqueue, acknowledge, lock, renew, history read/write, timer, and management operations:

• call count;
• error count with stable error category;
• duration histogram/buckets or sufficient aggregates for a host exporter;
• retry count.

Do not expose raw SQL, provider table names, connection details, instance IDs, or arbitrary exception messages as metric dimensions.

API requirements

• Keep the Provider / ProviderAdmin abstraction intact. PilotSwarm and other embedders must not query PostgreSQL or SQLite implementation tables.
• Define names, units, aggregation temporality, reset behavior, and optional-capability behavior.
• Make unsupported observations explicit rather than silently returning zero.
• Preserve provider-wide versus process-local scope in the type system or metadata.
• Use bounded enums/categories for operation, queue, result, and error type.
• Avoid requiring OpenTelemetry, Prometheus, or another backend in Duroxide core.
• Make snapshots cheap enough for periodic polling and safe with multiple runtimes/providers.

A possible shape is a versioned OperationalMetricsSnapshot containing typed gauges, monotonic counters, and histogram bucket snapshots plus scope/reset metadata. Exact API design is open.

Provider work

Implement the contract for the built-in SQLite provider and document the requirements for external providers. Coordinate with duroxide-pg for PostgreSQL implementation rather than making PilotSwarm depend on PostgreSQL tables.

Acceptance criteria

1. An embedder can obtain queue depth and oldest eligible age without provider-specific queries.
2. Lock/lease loss, timer lateness, dispatcher saturation, retries, replay cost, and provider-operation failures are observable through stable APIs.
3. Every field documents unit, scope, temporality, reset behavior, and unsupported-provider behavior.
4. Snapshot polling does not materially affect queue throughput.
5. SQLite tests cover populated, empty, contended, retry, timer-late, replay, and unsupported-capability cases.
6. Public APIs contain no provider implementation details or high-cardinality instance identifiers.
7. Existing SystemMetrics, QueueDepths, and MetricsSnapshot callers remain compatible or receive a documented migration path.
8. The Node binding can expose the resulting contract without reconstructing it from internal tables.

Relationship to PilotSwarm

PilotSwarm phase 1 will bridge the Duroxide APIs that already exist. These additions unblock phase 2: richer OTEL queue, lock, timer, replay, dispatcher, provider, and trace-correlated diagnostics.

No source files are named. Start by reviewing the existing SystemMetrics, QueueDepths, and MetricsSnapshot surfaces, then compare the quiescence/session-lease contract requested in #38. Define the provider-safe snapshot contract across the SQLite provider and Node binding, with documented scope, temporality, reset, and unsupported-capability behavior; completion requires the listed SQLite coverage and compatibility for existing callers.