Observability gap: no metric covers RPC receipt → history task durably committed

Is your feature request related to a problem? Please describe.

There is no metric measuring the interval from a task-producing RPC being received to the resulting history task being durably committed and eligible for dispatch.

RespondWorkflowTaskCompleted carries a bundle of Workflow Tasks and Activity Tasks that History must persist before Matching can dispatch them. Every segment of that path is instrumented. The transition from the RPC into the task queue is not — and the segments cannot be stitched into an end-to-end figure, because they are separate histograms across three services with no shared exemplar or trace context.

The practical effect: when a Worker completes a Workflow Task and the resulting Activity Task is slow to become pollable, no metric attributes the delay to the accept-and-persist step.

What already exists

Listing these explicitly so it's clear this isn't a request for something shipped. Descriptions quoted from common/metrics/metric_defs.go on main:

Metric	Description in metric_defs.go
task_latency_queue	"End-to-end latency for processing and completing a history task, from task generation to completion."
task_latency_load	"Latency from history task generation to loading into memory (persistence schedule to start latency)."
task_latency_schedule	"Latency from history task loading to start processing (in-memory schedule to start latency)."
task_latency_processing	"Latency for processing a history task one time."
task_latency	"Latency for processing and completing a history task. This latency is across all attempts but excludes any latencies related to workflow lock or user quota limit."

The load-bearing word is "generation". Every one of these begins after the task exists. None covers request received → task generated and committed.

On the RPC side, service_latency{operation="RespondWorkflowTaskCompleted"} covers the whole call including response serialisation, and persistence_latency{operation="UpdateWorkflowExecution"} covers only the database round trip. Neither isolates the accept-and-commit interval, and the difference between them is unattributed.

Measured evidence

Server 1.27.4, temporalio/auto-setup, single Worker under steady load. p99 over 3-minute windows:

Segment	Metric	p99
1 — Worker → Frontend → History persisted	service_latency{operation="RespondWorkflowTaskCompleted"}	9.89 ms
	persistence_latency{operation="UpdateWorkflowExecution"}	4.96 ms
2 — task generated → dispatched to Matching	task_latency_queue{operation="TransferActiveTaskActivity"}	85.85 ms
	├ task_latency_load	49.69 ms
	├ task_latency_schedule	0.99 ms
	└ task_latency_processing	9.23 ms
3 — Matching → poller	task_dispatch_latency{task_type="Activity"}	447.31 ms
	SDK temporal_activity_schedule_to_start_latency_seconds	460.10 ms

These are laptop-scale figures from a deliberately small deployment, included to show the shape — which segments are covered and which are not — rather than as representative production latencies. The close agreement between task_dispatch_latency (447 ms) and SDK-observed schedule-to-start (460 ms) is a useful check that segment 3 is measured correctly.

These cannot be summed. Three services, three histograms, no shared exemplar. Adding p99s does not produce a p99, so end-to-end acceptance latency is not derivable from what exists today.

Why it matters

When schedule-to-start rises, current signals distinguish "Matching is slow to dispatch" from "the task queue processor is behind" — but not "History was slow to accept and commit the task in the first place". That interval is invisible, and unlike the others it has no proxy to infer it from.

Note task_latency_load is already the largest contributor above (49.69 ms of 85.85 ms, 58%), and it's the segment that degrades under shard pressure. A new metric should sit alongside it rather than obscure it.

---

Describe the solution you'd like

A single histogram on the History service:

task_accepted_latency

• Measures: RPC receipt → resulting history task(s) durably committed and eligible for dispatch.
• Labels: at minimum namespace and task_category (transfer, timer, visibility), matching the existing task_latency_* family so it composes with them.
• Emitted from: History, at the point the transfer task is committed.

Deliberately scoped as one metric, not a new family — the sub-segments are already covered; only the leading interval is missing.

---

Describe alternatives you've considered

Trace context propagated through history tasks. This is the more complete answer, because it solves the stitching problem as well as the missing interval — you could follow one Workflow Task completion through Frontend, History and Matching. It is also a substantially larger change. If maintainers prefer this direction, the histogram above becomes unnecessary and this issue should be reframed.

Deriving it from existing metrics. Not possible, for the reason above: separate histograms across services cannot be composed into an end-to-end percentile.

Treating persistence_latency{operation="UpdateWorkflowExecution"} as sufficient. It covers only the database call, not the surrounding accept-and-commit work. If maintainers consider the remainder negligible, that's a reasonable resolution — but it would be worth stating explicitly, since it isn't apparent from the metric descriptions.

---

Additional context

Temporal Cloud exposure — possibly the higher-value half

I enumerated all 57 documented temporal_cloud_v1_* metrics from the OpenMetrics metrics reference. None of the task_latency_* family is exposed, nor task_dispatch_latency, nor task_schedule_to_start_latency.

The closest Cloud metrics are temporal_cloud_v1_service_latency_p99 (frontend RPC) and temporal_cloud_v1_approximate_backlog_count (queue depth) — one on either side of the entire history-task path, with nothing between.

So the gap is wider for Cloud customers than for self-hosted operators: not just the accept interval, but every segment between the RPC and the poller is unavailable. A Cloud customer seeing elevated schedule-to-start has no server-side metric to attribute it with.

If task_accepted_latency is added, exposing it — and ideally the existing task_latency_* family — via the Cloud OpenMetrics endpoint is what would make it useful to the customers most likely to hit this. Happy to split that into a separate issue if preferred.

Reproduction

Any self-hosted cluster with PROMETHEUS_ENDPOINT enabled, under load:

# Segment 1 — the RPC
histogram_quantile(0.99, sum by (le) (rate(
  service_latency_bucket{operation="RespondWorkflowTaskCompleted"}[3m])))

# Segment 2 — starts at task GENERATION, not at RPC receipt
histogram_quantile(0.99, sum by (le) (rate(
  task_latency_queue_bucket{operation="TransferActiveTaskActivity"}[3m])))

# Segment 3 — Matching to poller
histogram_quantile(0.99, sum by (le) (rate(
  task_dispatch_latency_bucket{task_type="Activity"}[3m])))

The gap is between the first and second queries.

Open questions

1. Is the accept-to-commit interval considered adequately covered by persistence_latency{operation="UpdateWorkflowExecution"}?
2. Is there an intentional reason task_latency_* is not exposed on Cloud — cardinality, or a deliberate abstraction boundary?
3. Would trace context propagation be preferred over a new histogram?

Related but distinct

• #3143 (per task queue metric naming) — different area
• #2435 (resource exhausted metrics) — different signal

Searched open and closed issues for task_accepted_latency, task_latency_queue, and RPC/task latency phrasings before filing; found no existing report of this gap.

Start with common/metrics/metric_defs.go and trace the History path from RPC receipt to the point where transfer tasks are committed. Check how existing task_latency_* metrics and labels are emitted, then verify the new histogram under load with the provided PromQL queries. Done means the accept-to-commit interval is measured with namespace and task_category labels without replacing existing segment metrics.