yesodweb / yesodweb/persistent
selectSource does not stream results
Nobody has claimed this yet.
- Dominant language
- Haskell
- Stars
- 486
- Forks
- 306
- PR merge metrics
- No merged PRs in 30d
Description
Currently, MySQL's backend doesn't actually stream the results: see this Google Groups thread, which I'm copying here for reference:
Background: I don't use persistent-mysql anymore but I wrote most of its
current code, so I still remember a bit or two.Currently, persistent-mysql will store in memory all of the query
results [1], even if you useselectSource. That's because it uses
mysql_store_result[2] instead ofmysql_use_result[3]. I can't
really remember right now the reason why I chose this way, but I do
remember that I initially tried usingmysql_use_resultwhich lead to
problems. In particular, note this paragraph from its documentation:On the other hand, you should not use mysql_use_result() if youare doing a lot of processing for each row on the client side,
or if the output is sent to a screen on which the user may type
a ^S (stop scroll). This ties up the server and prevent other
threads from updating any tables from which the data is being
fetched.We don't know how much processing is going to happen: it may be almost
zero (e.g., selectList), or it may be a lot (e.g., selectSource + fetch
a video linked on the entity and reencode it :). I faintly remember
seeing deadlocks, too, but I might be wrong.In sum: if using
persistent-mysql, all your queries are going to be
stored in memory regardless of the way you write your program.Cheers! =)
This is somewhat surprising, and it'd be nice to have documentation around that.
It would also be cool to have an unsafeSelectSource with documentation around limitations, possible problems, etc. that would use the msql_use_result function instead.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the persistent-mysql implementation of selectSource and its use of mysql_store_result. Review the mysql_use_result limitations described in the issue, then determine how an unsafeSelectSource API and accompanying documentation should address streaming, client-side processing, and connection blocking. Done means the behavior and trade-offs are documented and the proposed API is covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell, mysql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100