typelevel / typelevel/doobie

Batch updating a huge list causes OOM

Open
#466 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
2.2k
Forks
379
Avg merge
14m
Merged PRs (30d)
8

Description

 type Map = (String, Int)
 type Maps = Vector[Map]
 def insertMany(ms: Maps): ConnectionIO[Int] = {
   println(s"map size: ${ms.size}") //2312213
   val sql = "insert into collection (key, value) values (?, ?)"
   Update[Map](sql)
     .updateMany(ms)
 }

Above code causes OOM with memory settings: -Xms512m -Xmx1536m:

java.lang.OutOfMemoryError: Java heap space
    at scalaz.InvariantFunctor$class.$init$(InvariantFunctor.scala:46)
    at scalaz.FreeInstances$$anon$3.<init>(Free.scala:424)
    at scalaz.FreeInstances.freeMonad(Free.scala:424)
    at doobie.util.composite$LowerPriorityComposite$$anon$7$$anonfun$14.apply(composite.scala:119)
    at doobie.util.composite$LowerPriorityComposite$$anon$7$$anonfun$14.apply(composite.scala:119)
    at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
    at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
    at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
    at doobie.util.composite$LowerPriorityComposite$$anon$6$$anonfun$11.apply(composite.scala:109)
    at doobie.util.composite$LowerPriorityComposite$$anon$8$$anonfun$17.apply(composite.scala:129)
    at doobie.util.composite$LowerPriorityComposite$$anon$8$$anonfun$17.apply(composite.scala:129)
    at doobie.hi.preparedstatement$.set(preparedstatement.scala:222)
    at doobie.hi.preparedstatement$$anonfun$addBatchesAndExecute$1.apply(preparedstatement.scala:88)
    at doobie.hi.preparedstatement$$anonfun$addBatchesAndExecute$1.apply(preparedstatement.scala:88)
    at scala.collection.immutable.List$$anonfun$foldRight$1.apply(List.scala:397)
    at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
    at scala.collection.immutable.List.foldLeft(List.scala:84)
    at scala.collection.immutable.List.foldRight(List.scala:397)
    at doobie.hi.preparedstatement$.addBatchesAndExecute(preparedstatement.scala:88)
    at doobie.util.update$Update$class.updateMany(update.scala:113)
    at doobie.util.update$Update$$anon$5.updateMany(update.scala:198)
    at ...insertMany…

doobie-* versions:

[info] org.tpolecat:doobie-core_2.11:0.4.1
[info] org.tpolecat:doobie-hikari_2.11:0.4.1
[info] org.tpolecat:doobie-postgres_2.11:0.4.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading doobie.hi.preparedstatement.addBatchesAndExecute and doobie.util.update.Update.updateMany, especially the stack-trace locations preparedstatement.scala:88 and update.scala:113. Reproduce the two-million-row update with doobie 0.4.1 and inspect memory use during batch construction. Done means large updateMany calls no longer exhaust the configured heap, with coverage for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.