conductor-oss / conductor-oss/conductor

[PERF] `WorkflowMonitor` loads full definition catalog into memory, causing OOM at scale

Open
#1,006 0 comments 0 reactions 1 assignee Claimed by @manan164 View on GitHub
bug
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 5h
Merged PRs (30d)
41

Description

## Describe the bug

`WorkflowMonitor` periodically calls `metadataService.getWorkflowDefs()` to publish Prometheus metrics (`workflow_running`, `task_queue_depth`, `task_in_progress`). This executes:

```sql
SELECT json_data FROM meta_workflow_def ORDER BY name, version
```

Each row contains the full workflow definition (task lists, input/output mappings, failure workflows, etc.), but WorkflowMonitor only reads two fields: `name` and `ownerApp`. Everything else is deserialized and thrown away.

It also loads all versions of every definition, even though it groups by name and picks the latest version anyway (getPendingWorkflowToOwnerAppMap). The same pattern applies to task definitions — full `json_data` is loaded, but only `name`, `ownerApp`, and `concurrencyLimit` are used.

At ~100K definitions, this single call consumes hundreds of MB of heap and causes OutOfMemoryError on every reload cycle.

## Reload frequency
The reload happens every `metadata-refresh-interval` (default 10) × `stats.delay` (default 60s) = 10 minutes. With `stats.delay=30s`, it becomes every 5 minutes — causing a predictable 5-minute OOM restart loop.

## Why it affects all deployments
WorkflowMonitor is enabled by default (matchIfMissing = true), so it runs on every pod — including dedicated sweeper/async deployments that don't serve these metrics. The OOM cascades into unrelated operations (sweeper, task polling, DB connections) failing with heap exhaustion or HikariPool timeouts.

**Stack trace**
```
com.netflix.conductor.core.exception.NonTransientException: Java heap space
at com.netflix.conductor.postgres.dao.PostgresBaseDAO.getWithRetriedTransactions(PostgresBaseDAO.java:148)
at com.netflix.conductor.postgres.dao.PostgresBaseDAO.queryWithTransaction(PostgresBaseDAO.java:210)
at com.netflix.conductor.postgres.dao.PostgresMetadataDAO.getAllWorkflowDefs(PostgresMetadataDAO.java:212)
at com.netflix.conductor.service.MetadataServiceImpl.getWorkflowDefs(MetadataServiceImpl.java:169)
Caused by: java.lang.OutOfMemoryError: Java heap space
```

## Relevant code
```java
// WorkflowMonitor.java — loads full catalog
if (refreshCounter <= 0) {
workflowDefs = metadataService.getWorkflowDefs(); // all versions, full json_data
taskDefs = new ArrayList<>(metadataService.getTaskDefs());
refreshCounter = metadataRefreshInterval;
}

// But only uses name + ownerApp:
getPendingWorkflowToOwnerAppMap(workflowDefs)
```
## Proposed fix
**1.** Lightweight DAO method — fetch only the fields `WorkflowMonitor` actually needs instead of full `json_data`:
```
-- Workflows(Postgres): latest version only, minimal fields
SELECT DISTINCT ON (name) name, json_data::jsonb->>'ownerApp' as owner_app
FROM meta_workflow_def ORDER BY name, version DESC

-- Tasks: minimal fields
SELECT name, json_data::jsonb->>'ownerApp' as owner_app,
(json_data::jsonb->>'concurrencyLimit')::int as concurrency_limit
FROM meta_task_def
```
Reduces memory from hundreds of MB to a few KB regardless of definition count.

**Implementation notes:**

- Needs a new lightweight DTO (e.g. `WorkflowMetricInfo`) since the existing `WorkflowDefSummary` only has `name/version/createTime` — no `ownerApp` field.
- New methods on `MetadataDAO` interface (e.g. `getWorkflowMetricInfo()`, `getTaskMetricInfo()`) with a default fallback that projects from full defs, so persistence modules that don't override (Redis, Cassandra, SQLite) won't break:

``` java
default List getWorkflowMetricInfo() {
return getAllWorkflowDefsLatestVersions().stream()
.map(def -> new WorkflowMetricInfo(def.getName(), def.getOwnerApp()))
.collect(Collectors.toList());
}
```

- On the consumer side, `WorkflowMonitor` would hold lightweight DTOs instead of `List/List`, and `getPendingWorkflowToOwnerAppMap()` becomes unnecessary since the new query already returns one entry per workflow name.

**2.** At minimum, use `getAllWorkflowDefsLatestVersions()` — this method already exists and would cut 100K rows to ~1K (one per workflow name instead of one per version).

**3.** Default WorkflowMonitor to disabled (`matchIfMissing = false`) — or document `conductor.workflow-monitor.enabled=false` more prominently. Dedicated sweeper/async deployments have no use for these metrics.

## Workaround
Set `conductor.workflow-monitor.enabled=false` on deployments that don't need these metrics and increase heap on deployments where it stays enabled.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.