Add per-operation timing instrumentation for historically slow/deadlocked operations

Problem

Recent perf/deadlock regressions in the agent server were each caught by direct code review or a production incident rather than by an automated signal that says "this operation is taking too long / is stuck." For the operations that have historically wedged or slowed down, we have no per-operation timing that would let us (a) alert when one degrades, or (b) distinguish "slow" from "deadlocked."

We already emit coarse, privacy-bucketed conversation telemetry (see openhands-agent-server/openhands/agent_server/telemetry/), but it only captures whole-conversation duration_bucket and has no per-operation latency, so it cannot produce p50/p95 for the hot paths below.

Desired Behavior

Add per-operation timing instrumentation to the agent-server/SDK hot paths with a numeric latency property that can be percentiled (p50/p95/p99) in PostHog and, ideally, also emitted as OpenTelemetry/OpenInference spans. Timing should be recorded as wall-clock duration, and a "still in-flight / stuck" signal (or an explicit timeout) should be emitted so a deadlock is distinguishable from a merely slow operation.

Note: this is a deliberate change from the current bucketized/privacy-constrained telemetry policy (models.py / sanitizer.py), so it needs an explicit decision on how raw numeric durations are allowed through the allowlist without reintroducing a re-identification vector. Reuse the existing telemetry subscriber/sink plumbing rather than adding a parallel path.

Acceptance Criteria

Each operation below has previously been shown to deadlock or be very slow. Instrumentation must cover all of them, with a named latency metric per operation, and each historical failure must be reproducible as a failed/budget-exceeded threshold against a cheap fake-LLM or controlled-delay harness.

• Conversation lifecycle create/delete/close — global _lifecycle_lock wedge (#4514, fixed by #4570, tracked in #4569). Emit conversation_create, conversation_delete, and conversation_close latencies; a blocked create/close on one conversation must be visible (slow + no completion) without blocking measurement site.

• Event service load / runtime hydration — _get_or_load_event_service / _prepare_persisted_runtime disk I/O + runtime prep under the lifecycle lock (#4514, #4513, #4548). Emit event_service_load latency and a "blocked waiting for lifecycle lock" signal.

• Event search / bash event listing — event loop blocked by glob over an unbounded bash_events dir and scandir hot path (#4480, fixed by #4481). Emit event_search / bash_event_search latency incl. events-read count and directory fanout.

• ConversationInfo composition — GC wedge composing ConversationInfo off the event loop (#4417). Emit conversation_info_compose latency.

• LLM call concurrency / global config serialization — LLM requests unintentionally serialized through global config (#4473, OpenHands/OpenHands#16459). Emit per-LLM llm_call latency, queue/acquire time, and a concurrency count; the overlap (not serial) ratio must be measurable.

• switch_llm on the state lock — run-loop-held state lock deadlock (#3485, fixed by #3486). Emit switch_llm latency incl. state-lock wait time.

• stats_callback state-lock re-acquisition / stats streaming — ACP conversations hanging after prompt() on the state lock (#3348, fixed by #3349). Emit stats_streaming latency and state-lock wait time.

• subscribe_to_events initial-state push — wedged WS subscriber blocking registration (#3118, fixed by #3201). Emit subscribe_init_push latency (or timeout) and subscriber drain latency.

• ACP restart loopback secret lookup — async ACP restart loopback secret lookup deadlock (#3737). Emit acp_restart_secret_lookup latency.

• Model-info discovery / LLM construction probe — synchronous timeout-less model-info discovery hanging on unreachable/loopback endpoints (#3912). Emit model_info_discovery latency (bounded) and report timeout as a distinct terminal value.

• Conversation summary caching — unchanged summary recomputation (#4483). Emit summary_render latency with a cache-hit vs cache-miss dimension.

• Idle conversation eviction — idle conversation memory eviction (#4202). Emit conversation_evict latency and count.

• Lease renewal / autosave / pub/sub dispatch / condensation — historical perf work (#3168 lease renewal centralization, #3165 autosave batching, #3162 concurrent pub/sub, #3156 condensation forgotten-id set, #3163 file-store LRU, #3169 dedicated execution thread pool). Provide latency/count metrics for each so a regression that reintroduces serialization or O(N²) behavior is detectable.

• Latency totals/metrics are emitted as raw numeric durations (or p50/p95-friendly values) suitable for PostHog percentile aggregation, with the privacy trade-off documented and allowlisted explicitly.

• A "stuck"/timeout dimension is emitted so a deadlock (no completion) is distinguishable from a slow operation, rather than both looking identical in a latency histogram.

• Instrumentation lives behind the existing telemetry consent/kill-switch path (telemetry/policy.py) and is cheap enough (constant overhead) not to perturb the measured operations.

• Each historical case has a corresponding failing threshold in the pre-release load-test gate (see #4588) — i.e. the new metrics, not ad-hoc timing, are the assertion source where practical.

• Coverage of the operations above is verified by a test that forces each slow/deadlock scenario and asserts the metric is emitted with the expected terminal value (not just a happy-path duration).

References

• Deadlock/perf issues: #4514, #4569, #4570, #4513, #4548, #4480, #4481, #4417, #4473, #3485, #3486, #3348, #3349, #3118, #3201, #3737, #3912, #4483, #4202, #3168, #3165, #3162, #3156, #3163, #3169
• Existing telemetry plumbing: openhands-agent-server/openhands/agent_server/telemetry/ (models.py, sanitizer.py, subscriber.py, posthog_exporter.py, policy.py)
• Companion load-test gate: #4588

This issue was created by an AI agent (OpenHands) on behalf of the requester.

Start with the existing telemetry plumbing in openhands-agent-server/openhands/agent_server/telemetry/, especially models.py, sanitizer.py, subscriber.py, posthog_exporter.py, and policy.py. Then map the listed agent-server and SDK operations and review companion load-test gate #4588; done means every named operation emits the required latency and stuck/timeout signal, with privacy policy and threshold tests covering the historical cases.