SelfHealTemporalWorkflows drops status/type filters when paginating ListClosedWorkflowExecutions, sweeping every closed workflow in the namespace every 10s
- Langage dominant
- Python
- Étoiles
- 22.1k
- Forks
- 5.3k
- Métriques de merge des PR
- Métriques de PR en attente
Description
### Platform Version
v2.0.0 (self-managed, non-helm deployment; the code is unchanged on current `main`)
### What happened
`airbyte-cron`'s `SelfHealTemporalWorkflows` runs `TemporalClient.restartClosedWorkflowByStatus(FAILED)` every 10 seconds. `fetchClosedWorkflowsByStatus` builds the first `ListClosedWorkflowExecutionsRequest` with a `StatusFilter` and a `WorkflowTypeFilter`, but the requests built inside the pagination loop carry only the namespace and the page token, so both filters are dropped from page 2 onward:
```kotlin
var workflowExecutionsRequest =
ListClosedWorkflowExecutionsRequest.newBuilder()
.setNamespace(workflowClientWrapped.namespace)
.setStatusFilter(StatusFilter.newBuilder().setStatus(executionStatus).build())
.setTypeFilter(WorkflowTypeFilter.newBuilder().setName(ConnectionManagerWorkflow::class.java.getSimpleName()).build())
.build()
do {
// ...
workflowExecutionsRequest =
ListClosedWorkflowExecutionsRequest.newBuilder()
.setNamespace(workflowClientWrapped.namespace)
.setNextPageToken(token) // StatusFilter and TypeFilter are gone here
.build()
} while (token != null && token.size() > 0)
```
Permalink: https://github.com/airbytehq/airbyte-platform/blob/c645707cc8d0fdf3aaa7c741ce32eb89693d525b/airbyte-commons-temporal/src/main/kotlin/io/airbyte/commons/temporal/TemporalClient.kt (`fetchClosedWorkflowsByStatus`)
Consequences:
1. As soon as a single FAILED `ConnectionManagerWorkflow` record exists within the retention window, every 10-second tick paginates through **all** closed workflow executions in the namespace, not just failed connection-manager workflows. `refreshRunningWorkflow` additionally does a full unfiltered `ListOpenWorkflowExecutions` sweep on every tick.
2. On SQL visibility stores the sweep is quadratic: Postgres executes Temporal's keyset-pagination predicate as a row filter rather than an index bound, so page N re-scans everything above it. Cost grows O(N^2) with the number of closed workflows.
3. The FAILED visibility record keeps re-arming the full sweep for the entire retention period, even after the workflow was successfully restarted, because the closed execution record remains in visibility.
### Impact observed
Self-hosted deployment against an external Temporal cluster whose `default` namespace held ~600k closed executions (7-day retention):
- sustained ~4.8 `ListClosedWorkflowExecutions` req/s, 24/7, capped only by Temporal's namespace rate limit
- the visibility Postgres (4 vCPU) pinned at ~48% CPU for weeks (~1.7 average active sessions; ~522k visibility index tuples read per query; list p95 ~1s)
- load dropped to ~3% the moment the last FAILED record aged out of retention, and re-armed the instant one connection failed again
This is not only a shared-cluster problem: any deployment with many closed workflow records and an occasional connection failure pays this cost. It is very likely the root cause of the recurring `RESOURCE_EXHAUSTED: namespace rate limit exceeded` errors from `SelfHealTemporalWorkflows` (#66717, #30691) — the stack trace in #66717 points at exactly this pagination loop (`TemporalClient.fetchClosedWorkflowsByStatus`).
### Suggested fix
Carry the filters into the paginated requests, e.g. rebuild via `workflowExecutionsRequest.toBuilder().setNextPageToken(token).build()` instead of a fresh builder with only namespace and token. Longer term, migrating to the non-deprecated `ListWorkflowExecutions` API with a query filter (`ExecutionStatus = 'Failed' AND WorkflowType = 'ConnectionManagerWorkflow'`) pushes the filtering server-side and eliminates the sweep entirely.
Related: #18853 — a configurable Temporal namespace would reduce the blast radius on shared clusters, but the filter drop hurts single-tenant deployments too.
### Are you willing to submit a PR?
Yes, the fix is small, but my understanding is that platform contributions are currently not accepted. Happy to submit one if maintainers want it.
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13086
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.