conductor-oss / conductor-oss/conductor

Postgres tsvector limit causes workflow Failures and resource exhaustion

Open
#604 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d
Merged PRs (30d)
33

Description

### Describe the bug

When running Conductor with Postgres, a task that has both large input and output can exceed a limit, causing a decision error and preventing the workflow from completing.

The error thrown is: `error: string is too long for tsvector (1090676 bytes, max 1048575 bytes)`.
Conductor will repeatedly attempt to process this workflow, which leads to increased resource usage on Postgres.

In our test environment, this causes Postgres to use approximately 0.5 CPU core for each failing workflow, along with a gradual increase in memory usage until Postgres starts writing to disk.
This rapidly fills up disk space, eventually causing Postgres to crash when the volume is full.

These metrics are from just 10 problematic workflows.
Image
Image
Image

---

We tried lowering the task input and output limit to 1MB, but the issue can still be triggered because `jsonb_to_tsvector` is used on the column that combines input and output.
You can also trigger the error with an input smaller than 1MB.

For example, an output of 907,321 bytes will still trigger the error:
```javascript
{id: Array.from({length: 50_000}, (_, i) => Math.random() * 10_000_000)}
```

### Details
Conductor version: 3.x
Persistence implementation: Postgres
Queue implementation: Redis
Lock: Redis

### Steps to reproduce

1. Create a workflow using a task.
2. Have the task output the example output above.
3. The workflow will become stuck and the error will be present in the logs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.