flowable / flowable/flowable-engine

Endless job processing loop when using JMS/ActiveMQ

Open
#2,442 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
7h 8m
Merged PRs (30d)
2

Description

**Describe the bug**

When using JMS/ActiveMQ as per https://flowable.com/open-source/docs/bpmn/ch18-Advanced/#message-queue-based-async-executor, the Jobs can get into an endless lock fail / retry loop.

This initially surfaced as a lock failure described in this forum thread https://forum.flowable.org/t/update-on-act-ru-execution/4949. After deeper investigation, the traceback included

```
at org.flowable.common.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.flowable.common.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:56)
at org.flowable.common.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:51)
at org.flowable.job.service.impl.asyncexecutor.ExecuteAsyncRunnable.lockJob(ExecuteAsyncRunnable.java:190)
at org.flowable.job.service.impl.asyncexecutor.ExecuteAsyncRunnable.run(ExecuteAsyncRunnable.java:113)
at org.flowable.spring.executor.jms.JobMessageListener.onMessage(JobMessageListener.java:40)
```

And with a little debug logging we are able to track this down to a single job ID that retries 100s of times per second.

**Expected behavior**

The job should (eventually) fail and be put into the `ACT_RU_DEADLETTER_JOB` table.

**Code**

We use the example https://flowable.com/open-source/docs/bpmn/ch18-Advanced/#message-queue-based-async-executor to configure our Async JMS integration.

**Additional context**

Flowable Version : 6.5.0
Flowable in a Spring container, but NOT Spring Boot.
ActiveMQ

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ExecuteAsyncRunnable.lockJob and JobMessageListener.onMessage, using the message-queue-based async executor configuration linked in the issue. Reproduce the lock-failure retry loop with JMS/ActiveMQ and inspect how failed jobs are retried. Done means a persistently failing job eventually appears in ACT_RU_DEADLETTER_JOB instead of retrying indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.