conductor-oss / conductor-oss/conductor
Sometimes forked tasks do not execute due to incorrect queue count
- Dominant language
- Java
- Stars
- 32.2k
- Forks
- 1k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 37
Description
**Describe the bug**
I have noticed the following bug shown in the video.
https://github.com/user-attachments/assets/7e982eb4-acf8-479c-94d6-749ff90ec606
In video, we can observe that when first time workflow is run, some of the forked tasks get stuck. Immediately after that, I ran same workflow which executes immediately.
In my use case application polls tasks first by calling 'all' endpoint which returns number of scheduled tasks in queue for each registered task. Then task is polled if count is greater than 0.
I have written C# console program which behaves similarly.
C# application is available [here](https://github.com/boma96/ConductorPollingTest)
Every 200ms task calls 'all' endpoint and prints the number of scheduled tasks for `TEST_worker` task.
For the video above I have observed following logs:
```
0
0
8
Polled TEST_worker
Completed TEST_worker
11
Polled TEST_worker
Completed TEST_worker
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
Polled TEST_worker
Completed TEST_worker
11
Polled TEST_worker
Completed TEST_worker
10
Polled TEST_worker
Completed TEST_worker
9
Polled TEST_worker
Completed TEST_worker
8
Polled TEST_worker
Completed TEST_worker
7
Polled TEST_worker
Completed TEST_worker
6
Polled TEST_worker
Completed TEST_worker
5
Polled TEST_worker
Completed TEST_worker
4
Polled TEST_worker
Completed TEST_worker
3
Polled TEST_worker
Completed TEST_worker
2
Polled TEST_worker
Completed TEST_worker
1
Polled TEST_worker
Completed TEST_worker
0
```
Notice the queue count is incorrect for first case. Even though we scheduled 12 tasks for execution only 2 of them are executed. Others are stuck in IN_PROGRESS state. Immediately after that workflow is run once again and we can see that queue count decreases from 12 to 0 as expected.
The issue happens at random. For the most part, workflow executes just fine.
**Details**
Conductor version: Built from the main branch.
Persistence implementation: Postgres
Queue implementation: Postgres
Lock: Not sure, I am using docker-compose.postgres.yaml to start Conductor, don't see redis or zookeeper containers.
**To Reproduce**
Steps to reproduce the behavior:
1. Start Conductor using docker-compose-postgres.yaml
2. Start the C# application
3. Start workflows, keep starting until you observe an issue.
**Expected behavior**
All of the forked tasks execute as expected.
Contributor guide
Assessment
This issue has not been assessed yet.