JetBrains / JetBrains/Exposed

Memory leak in long running transaction

Open
#2,407 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Kotlin
Stars
9.3k
Forks
798
Avg merge
4d 2h
Merged PRs (30d)
26

Description

Hi,

Using pgsql & suspended transactions in dsl mode, we have a long running transaction with a loop, e.g.

```kotlin
transaction {
var doContinue = true
while (doContinue) {
val toProcess = getDbElementsToProcess()
processElements(toProcess)
if (toProcess.isEmpty()) doContinue = false
}
}
```

If we have a huge number of elements to process (> 100 000) and the transactions lasts few minutes, we can eventually have an Out Of Memory. Looking in VisualVM, we see this is caused by multiple PreparedStatements that are never garbage collected.

Yet if we open a new transaction on each loop run, we don't have troubles anymore. e.g.

```kotlin
var doContinue = true
while (doContinue) {
transaction {
val toProcess = getDbElementsToProcess()
processElements(toProcess)
if (toProcess.isEmpty()) doContinue = false
}
}
```

From another post, we understood that closing the transaction does close all prepared statements and free the memory.

However we'd like to avoid having to open a new transaction on each loop run. Is there any way to close prepared statements properly, optionally manually?

Contributor guide

Open the contributing guide

Research direction

Start with the suspended-transaction DSL and the PreparedStatement lifecycle described in the report, using the provided long-running loop to reproduce the leak. Use VisualVM to confirm statement retention, then verify that prepared statements are released without requiring a new transaction for every loop iteration.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin, postgresql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.