finos / finos/fluxnova-plugins

Process variable overflow after multiple agentic loop iterations

Open
#33 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3
Forks
7
Avg merge
5d 4h
Merged PRs (30d)
3

Description

**Description:**
When an agentic loop runs for multiple iterations, the accumulated conversation history stored as a process variable exceeds Fluxnova's 4000-character limit for the `TEXT_` column in `ACT_HI_DETAIL`, causing the process to fail and roll back.

**Steps to Reproduce:**
Start a process that invokes the agentic loop
Set a system prompt that cannot resolve (forcing multiple iterations)
Observe failure after N iterations as the variable grows beyond 4000 chars ( 4-5 iterations should do it)

**Expected Behaviour:**
Process continues executing across iterations without database constraint violations.

**Actual Behaviour:**
`JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)"` — process rolls back, 2 prior sub-executors also rolled back.

**Error Output:**
```
org.finos.fluxnova.bpm.engine.ProcessEnginePersistenceException: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace.
at org.finos.fluxnova.bpm.engine.impl.util.ExceptionUtil.wrapPersistenceException(ExceptionUtil.java:340)
at org.finos.fluxnova.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:138)
at org.finos.fluxnova.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:361)
at org.finos.fluxnova.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:323)
at org.finos.fluxnova.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:295)
at org.finos.fluxnova.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:267)
at org.finos.fluxnova.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:188)
at org.finos.fluxnova.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:120)
at org.finos.fluxnova.bpm.engine.spring.SpringTransactionInterceptor.lambda$execute$0(SpringTransactionInterceptor.java:62)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.finos.fluxnova.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:62)
at org.finos.fluxnova.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.finos.fluxnova.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:34)
at org.finos.fluxnova.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.finos.fluxnova.bpm.engine.impl.interceptor.ExceptionCodeInterceptor.execute(ExceptionCodeInterceptor.java:55)
at org.finos.fluxnova.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:57)
at org.finos.fluxnova.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:110)
at org.finos.fluxnova.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:71)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.finos.fluxnova.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation 'INSERT HistoricVariableUpdateEventEntity[abe82ed2-6b03-11f1-a03b-00155dcd0604]' with message '
### Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.finos.fluxnova.bpm.engine.impl.persistence.entity.HistoricDetailEntity.insertHistoricVariableUpdateEvent (batch index #3) failed. 2 prior sub executor(s) completed successfully, but will be rolled back. Cause: org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.finos.fluxnova.bpm.engine.impl.persistence.entity.HistoricDetailEntity.insertHistoricVariableUpdateEvent (batch index #3) failed. 2 prior sub executor(s) completed successfully, but will be rolled back. Cause: org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
org.h2.jdbc.JdbcSQLDataException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
'. Flush summary:
[
INSERT HistoricVariableInstanceEntity[aa87b46f-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableInstanceEntity[abe855e5-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricJobLogEventEntity[abe87cfa-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableUpdateEventEntity[aa878d5e-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableUpdateEventEntity[aa87b470-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableUpdateEventEntity[aa87db81-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableUpdateEventEntity[abe82ed2-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableUpdateEventEntity[abe855e3-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableUpdateEventEntity[abe855e6-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricVariableUpdateEventEntity[abe87cf9-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricTaskInstanceEventEntity[abe855e8-6b03-11f1-a03b-00155dcd0604]
INSERT HistoricActivityInstanceEventEntity[Activity_03s612r:abe855e7-6b03-11f1-a03b-00155dcd0604]
INSERT ExecutionEntity[abe855e4-6b03-11f1-a03b-00155dcd0604]
INSERT TaskEntity[abe855e8-6b03-11f1-a03b-00155dcd0604]
INSERT VariableInstanceEntity[abe855e5-6b03-11f1-a03b-00155dcd0604]
DELETE MessageEntity[a70d5a9d-6b03-11f1-a03b-00155dcd0604]
UPDATE VariableInstanceEntity[9e984875-6b03-11f1-a03b-00155dcd0604]
UPDATE VariableInstanceEntity[9e986f87-6b03-11f1-a03b-00155dcd0604]
UPDATE VariableInstanceEntity[9e98e4ce-6b03-11f1-a03b-00155dcd0604]
DELETE_BULK deleteByteArrayNoRevisionCheck a91e123e-6b03-11f1-a03b-00155dcd0604
UPDATE ExecutionEntity[9ce08651-6b03-11f1-a03b-00155dcd0604]
UPDATE HistoricVariableInstanceEntity[9e984875-6b03-11f1-a03b-00155dcd0604]
UPDATE HistoricVariableInstanceEntity[9e98e4ce-6b03-11f1-a03b-00155dcd0604]
]
at org.finos.fluxnova.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:137)
... 19 more
Caused by: org.apache.ibatis.exceptions.PersistenceException:
### Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.finos.fluxnova.bpm.engine.impl.persistence.entity.HistoricDetailEntity.insertHistoricVariableUpdateEvent (batch index #3) failed. 2 prior sub executor(s) completed successfully, but will be rolled back. Cause: org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.finos.fluxnova.bpm.engine.impl.persistence.entity.HistoricDetailEntity.insertHistoricVariableUpdateEvent (batch index #3) failed. 2 prior sub executor(s) completed successfully, but will be rolled back. Cause: org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.flushStatements(DefaultSqlSession.java:254)
at org.finos.fluxnova.bpm.engine.impl.db.sql.DbSqlSession.flushBatchOperations(DbSqlSession.java:386)
at org.finos.fluxnova.bpm.engine.impl.db.sql.BatchDbSqlSession.executeDbOperations(BatchDbSqlSession.java:73)
at org.finos.fluxnova.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:341)
... 18 more
Caused by: org.apache.ibatis.executor.BatchExecutorException: org.finos.fluxnova.bpm.engine.impl.persistence.entity.HistoricDetailEntity.insertHistoricVariableUpdateEvent (batch index #3) failed. 2 prior sub executor(s) completed successfully, but will be rolled back. Cause: org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:148)
at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:129)
at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:122)
at org.apache.ibatis.executor.CachingExecutor.flushStatements(CachingExecutor.java:115)
at org.apache.ibatis.session.defaults.DefaultSqlSession.flushStatements(DefaultSqlSession.java:252)
... 21 more
Caused by: org.h2.jdbc.JdbcBatchUpdateException: Value too long for column "TEXT_ CHARACTER VARYING(4000)": "U&'[{""role"":""ASSISTANT"",""content"":""Let''s begin the analysis of the flagged tran... (4372)"; SQL statement:
insert into ACT_HI_DETAIL
(
ID_,
TYPE_,
PROC_DEF_KEY_,
PROC_DEF_ID_,
ROOT_PROC_INST_ID_,
PROC_INST_ID_,
EXECUTION_ID_,
ACT_INST_ID_,
CASE_DEF_KEY_,
CASE_DEF_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
NAME_,
REV_,
VAR_INST_ID_,
VAR_TYPE_,
TIME_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
SEQUENCE_COUNTER_,
TENANT_ID_,
OPERATION_ID_,
REMOVAL_TIME_,
INITIAL_
)
values (
?,
'VariableUpdate',
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
) [22001-232]
at org.h2.jdbc.JdbcPreparedStatement.executeBatch(JdbcPreparedStatement.java:1277)
at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:126)
... 25 more

```
**Root Cause:**
Conversation history is being stored as a String type variable. Fluxnova routes String variables through StringType serializer, which has a hard 4000-char column limit. After sufficient iterations the variable breaches this.

**Possible Solutions:**
Serialize the variable as a JSON object (ByteArrayType) — routes to ACT_GE_BYTEARRAY, no size limit. Audit trail impact and readability of audit trail in Control Centre TBD.

@tom-stavert @mr-dibbs @finos/fluxnova-maintainers thoughts please.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the agentic loop with a prompt that forces multiple iterations, then trace HistoricDetailEntity.insertHistoricVariableUpdateEvent and the ACT_HI_DETAIL TEXT_ column shown in the stack trace. Done means the process completes multiple iterations without a database constraint violation or rollback.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.