conductor-oss / conductor-oss/conductor

Sometimes forked tasks do not execute due to incorrect queue count

Open
#282 0 comments 4 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 3h
Merged PRs (30d)
37

Description

**Describe the bug**
I have noticed the following bug shown in the video.

https://github.com/user-attachments/assets/7e982eb4-acf8-479c-94d6-749ff90ec606

In video, we can observe that when first time workflow is run, some of the forked tasks get stuck. Immediately after that, I ran same workflow which executes immediately.

In my use case application polls tasks first by calling 'all' endpoint which returns number of scheduled tasks in queue for each registered task. Then task is polled if count is greater than 0.

I have written C# console program which behaves similarly.
C# application is available [here](https://github.com/boma96/ConductorPollingTest)
Every 200ms task calls 'all' endpoint and prints the number of scheduled tasks for `TEST_worker` task.

For the video above I have observed following logs:

```
0
0
8
Polled TEST_worker
Completed TEST_worker
11
Polled TEST_worker
Completed TEST_worker
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
Polled TEST_worker
Completed TEST_worker
11
Polled TEST_worker
Completed TEST_worker
10
Polled TEST_worker
Completed TEST_worker
9
Polled TEST_worker
Completed TEST_worker
8
Polled TEST_worker
Completed TEST_worker
7
Polled TEST_worker
Completed TEST_worker
6
Polled TEST_worker
Completed TEST_worker
5
Polled TEST_worker
Completed TEST_worker
4
Polled TEST_worker
Completed TEST_worker
3
Polled TEST_worker
Completed TEST_worker
2
Polled TEST_worker
Completed TEST_worker
1
Polled TEST_worker
Completed TEST_worker
0
```

Notice the queue count is incorrect for first case. Even though we scheduled 12 tasks for execution only 2 of them are executed. Others are stuck in IN_PROGRESS state. Immediately after that workflow is run once again and we can see that queue count decreases from 12 to 0 as expected.

The issue happens at random. For the most part, workflow executes just fine.

**Details**
Conductor version: Built from the main branch.
Persistence implementation: Postgres
Queue implementation: Postgres
Lock: Not sure, I am using docker-compose.postgres.yaml to start Conductor, don't see redis or zookeeper containers.

**To Reproduce**
Steps to reproduce the behavior:
1. Start Conductor using docker-compose-postgres.yaml
2. Start the C# application
3. Start workflows, keep starting until you observe an issue.

**Expected behavior**
All of the forked tasks execute as expected.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.