airbytehq / airbytehq/airbyte

SelfHealTemporalWorkflows drops status/type filters when paginating ListClosedWorkflowExecutions, sweeping every closed workflow in the namespace every 10s

Ouverte
#81,631 6 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
autoteam community team/use
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

### Platform Version

v2.0.0 (self-managed, non-helm deployment; the code is unchanged on current `main`)

### What happened

`airbyte-cron`'s `SelfHealTemporalWorkflows` runs `TemporalClient.restartClosedWorkflowByStatus(FAILED)` every 10 seconds. `fetchClosedWorkflowsByStatus` builds the first `ListClosedWorkflowExecutionsRequest` with a `StatusFilter` and a `WorkflowTypeFilter`, but the requests built inside the pagination loop carry only the namespace and the page token, so both filters are dropped from page 2 onward:

```kotlin
var workflowExecutionsRequest =
ListClosedWorkflowExecutionsRequest.newBuilder()
.setNamespace(workflowClientWrapped.namespace)
.setStatusFilter(StatusFilter.newBuilder().setStatus(executionStatus).build())
.setTypeFilter(WorkflowTypeFilter.newBuilder().setName(ConnectionManagerWorkflow::class.java.getSimpleName()).build())
.build()

do {
// ...
workflowExecutionsRequest =
ListClosedWorkflowExecutionsRequest.newBuilder()
.setNamespace(workflowClientWrapped.namespace)
.setNextPageToken(token) // StatusFilter and TypeFilter are gone here
.build()
} while (token != null && token.size() > 0)
```

Permalink: https://github.com/airbytehq/airbyte-platform/blob/c645707cc8d0fdf3aaa7c741ce32eb89693d525b/airbyte-commons-temporal/src/main/kotlin/io/airbyte/commons/temporal/TemporalClient.kt (`fetchClosedWorkflowsByStatus`)

Consequences:

1. As soon as a single FAILED `ConnectionManagerWorkflow` record exists within the retention window, every 10-second tick paginates through **all** closed workflow executions in the namespace, not just failed connection-manager workflows. `refreshRunningWorkflow` additionally does a full unfiltered `ListOpenWorkflowExecutions` sweep on every tick.
2. On SQL visibility stores the sweep is quadratic: Postgres executes Temporal's keyset-pagination predicate as a row filter rather than an index bound, so page N re-scans everything above it. Cost grows O(N^2) with the number of closed workflows.
3. The FAILED visibility record keeps re-arming the full sweep for the entire retention period, even after the workflow was successfully restarted, because the closed execution record remains in visibility.

### Impact observed

Self-hosted deployment against an external Temporal cluster whose `default` namespace held ~600k closed executions (7-day retention):

- sustained ~4.8 `ListClosedWorkflowExecutions` req/s, 24/7, capped only by Temporal's namespace rate limit
- the visibility Postgres (4 vCPU) pinned at ~48% CPU for weeks (~1.7 average active sessions; ~522k visibility index tuples read per query; list p95 ~1s)
- load dropped to ~3% the moment the last FAILED record aged out of retention, and re-armed the instant one connection failed again

This is not only a shared-cluster problem: any deployment with many closed workflow records and an occasional connection failure pays this cost. It is very likely the root cause of the recurring `RESOURCE_EXHAUSTED: namespace rate limit exceeded` errors from `SelfHealTemporalWorkflows` (#66717, #30691) — the stack trace in #66717 points at exactly this pagination loop (`TemporalClient.fetchClosedWorkflowsByStatus`).

### Suggested fix

Carry the filters into the paginated requests, e.g. rebuild via `workflowExecutionsRequest.toBuilder().setNextPageToken(token).build()` instead of a fresh builder with only namespace and token. Longer term, migrating to the non-deprecated `ListWorkflowExecutions` API with a query filter (`ExecutionStatus = 'Failed' AND WorkflowType = 'ConnectionManagerWorkflow'`) pushes the filtering server-side and eliminates the sweep entirely.

Related: #18853 — a configurable Temporal namespace would reduce the blast radius on shared clusters, but the filter drop hurts single-tenant deployments too.

### Are you willing to submit a PR?

Yes, the fix is small, but my understanding is that platform contributions are currently not accepted. Happy to submit one if maintainers want it.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13086

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.