typelevel / typelevel/doobie

Streaming does not work for MySQL

Open
#715 7 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
2.2k
Forks
379
Avg merge
14m
Merged PRs (30d)
8

Description

I wrote the following code

val xs = Transactor.fromDriverManager[IO](
           "com.mysql.cj.jdbc.Driver", 
           "jdbc:mysql://myserver:3306/mydb?useLegacyDatetimeCode=false&nullNamePatternMatchesAll=true&serverTimezone=America/Chicago&useSSL=false", 
           "user", 
           "pass")

sql"select id from foo"
       .query[Long]
       .stream
       .take(10)
       .compile
       .toList
       .transact(xs)
       .unsafeRunSync
       .foreach(println)

Here the table foo has 100 million records. when I run this code I get an error

java.lang.OutOfMemoryError: Java heap space

So even though I am using Streaming, the program is trying to load all the IDs into memory and that is why it is getting a out of memory exception.

In the past, I tried to do Streaming of query results using Slick and I faced similar issue there as well. It turned out that in order to enable Streaming on MySQL. we need a special setup. This was documented here

I want to know what is the right syntax in Doobie so that we can stream results from MySQL as opposed to loading the entire resultset in memory.

I have posted this problem here as well https://stackoverflow.com/questions/50008051/doobie-streaming-not-working-for-mysql

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported Transactor.fromDriverManager setup and the .stream call, then compare the MySQL JDBC streaming requirements documented in Slick issue 1218. Reproduce the query against a large table and verify that taking 10 rows does not load the complete result set into memory; the Stack Overflow report provides additional context.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, scala
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.