Batch updating a huge list causes OOM
Open
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 2.2k
- Forks
- 379
- Avg merge
- 14m
- Merged PRs (30d)
- 8
Description
type Map = (String, Int)
type Maps = Vector[Map]
def insertMany(ms: Maps): ConnectionIO[Int] = {
println(s"map size: ${ms.size}") //2312213
val sql = "insert into collection (key, value) values (?, ?)"
Update[Map](sql)
.updateMany(ms)
}
Above code causes OOM with memory settings: -Xms512m -Xmx1536m:
java.lang.OutOfMemoryError: Java heap space
at scalaz.InvariantFunctor$class.$init$(InvariantFunctor.scala:46)
at scalaz.FreeInstances$$anon$3.<init>(Free.scala:424)
at scalaz.FreeInstances.freeMonad(Free.scala:424)
at doobie.util.composite$LowerPriorityComposite$$anon$7$$anonfun$14.apply(composite.scala:119)
at doobie.util.composite$LowerPriorityComposite$$anon$7$$anonfun$14.apply(composite.scala:119)
at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
at doobie.util.composite$LowerPriorityComposite$$anon$8$$anonfun$17.apply(composite.scala:129)
at doobie.util.composite$LowerPriorityComposite$$anon$8$$anonfun$17.apply(composite.scala:129)
at doobie.hi.preparedstatement$.set(preparedstatement.scala:222)
at doobie.hi.preparedstatement$$anonfun$addBatchesAndExecute$1.apply(preparedstatement.scala:88)
at doobie.hi.preparedstatement$$anonfun$addBatchesAndExecute$1.apply(preparedstatement.scala:88)
at scala.collection.immutable.List$$anonfun$foldRight$1.apply(List.scala:397)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at scala.collection.immutable.List.foldRight(List.scala:397)
at doobie.hi.preparedstatement$.addBatchesAndExecute(preparedstatement.scala:88)
at doobie.util.update$Update$class.updateMany(update.scala:113)
at doobie.util.update$Update$$anon$5.updateMany(update.scala:198)
at ...insertMany…
doobie-* versions:
[info] org.tpolecat:doobie-core_2.11:0.4.1
[info] org.tpolecat:doobie-hikari_2.11:0.4.1
[info] org.tpolecat:doobie-postgres_2.11:0.4.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading doobie.hi.preparedstatement.addBatchesAndExecute and doobie.util.update.Update.updateMany, especially the stack-trace locations preparedstatement.scala:88 and update.scala:113. Reproduce the two-million-row update with doobie 0.4.1 and inspect memory use during batch construction. Done means large updateMany calls no longer exhaust the configured heap, with coverage for the reported case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100