Streaming does not work for MySQL
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 2.2k
- Forks
- 379
- Avg merge
- 14m
- Merged PRs (30d)
- 8
Description
I wrote the following code
val xs = Transactor.fromDriverManager[IO](
"com.mysql.cj.jdbc.Driver",
"jdbc:mysql://myserver:3306/mydb?useLegacyDatetimeCode=false&nullNamePatternMatchesAll=true&serverTimezone=America/Chicago&useSSL=false",
"user",
"pass")
sql"select id from foo"
.query[Long]
.stream
.take(10)
.compile
.toList
.transact(xs)
.unsafeRunSync
.foreach(println)
Here the table foo has 100 million records. when I run this code I get an error
java.lang.OutOfMemoryError: Java heap space
So even though I am using Streaming, the program is trying to load all the IDs into memory and that is why it is getting a out of memory exception.
In the past, I tried to do Streaming of query results using Slick and I faced similar issue there as well. It turned out that in order to enable Streaming on MySQL. we need a special setup. This was documented here
I want to know what is the right syntax in Doobie so that we can stream results from MySQL as opposed to loading the entire resultset in memory.
I have posted this problem here as well https://stackoverflow.com/questions/50008051/doobie-streaming-not-working-for-mysql
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported Transactor.fromDriverManager setup and the .stream call, then compare the MySQL JDBC streaming requirements documented in Slick issue 1218. Reproduce the query against a large table and verify that taking 10 rows does not load the complete result set into memory; the Stack Overflow report provides additional context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, scala
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100