yesodweb / yesodweb/persistent

selectSource does not stream results

Open
#657 11 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

MySQL Postgres
Dominant language
Haskell
Stars
486
Forks
306
PR merge metrics
No merged PRs in 30d

Description

Currently, MySQL's backend doesn't actually stream the results: see this Google Groups thread, which I'm copying here for reference:

Background: I don't use persistent-mysql anymore but I wrote most of its
current code, so I still remember a bit or two.

Currently, persistent-mysql will store in memory all of the query
results [1], even if you use selectSource. That's because it uses
mysql_store_result [2] instead of mysql_use_result [3]. I can't
really remember right now the reason why I chose this way, but I do
remember that I initially tried using mysql_use_result which lead to
problems. In particular, note this paragraph from its documentation:

On the other hand, you should not use mysql_use_result() if you 

are doing a lot of processing for each row on the client side,
or if the output is sent to a screen on which the user may type
a ^S (stop scroll). This ties up the server and prevent other
threads from updating any tables from which the data is being
fetched.

We don't know how much processing is going to happen: it may be almost
zero (e.g., selectList), or it may be a lot (e.g., selectSource + fetch
a video linked on the entity and reencode it :). I faintly remember
seeing deadlocks, too, but I might be wrong.

In sum: if using persistent-mysql, all your queries are going to be
stored in memory regardless of the way you write your program.

Cheers! =)

This is somewhat surprising, and it'd be nice to have documentation around that.

It would also be cool to have an unsafeSelectSource with documentation around limitations, possible problems, etc. that would use the msql_use_result function instead.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the persistent-mysql implementation of selectSource and its use of mysql_store_result. Review the mysql_use_result limitations described in the issue, then determine how an unsafeSelectSource API and accompanying documentation should address streaming, client-side processing, and connection blocking. Done means the behavior and trade-offs are documented and the proposed API is covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell, mysql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.