haskell-servant / haskell-servant/servant
SourceT's readFile uses an innefficient chunk size
- Dominant language
- Haskell
- Stars
- 2k
- Forks
- 427
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 5
Description
4096 is not the most optimal chunk size, and so this function is slower at reading files than it could be.
In a quick benchmark, using the example from https://docs.servant.dev/en/stable/cookbook/basic-streaming/Streaming.html, and a 1 gb "README.md" file, curl to /dev/null takes 1.84 seconds.
I was able to optimize that to 0.68 seconds. My optimisation happened to read it into a lazy ByteString and then stream out the toChunks, which is ugly and I don't recommend. (I was actually just investigating how to stream a lazy ByteString through servant when I noticed this performance problem.)
Probably the right fix is to change 4096 to some more optimal chunk size. Data.ByteString.Lazy.Internal.defaultChunkSize is a good one.
Contributor guide
Research direction
Locate SourceT's readFile implementation and inspect how it chooses the 4096-byte chunk size. Compare the current behavior with Data.ByteString.Lazy.Internal.defaultChunkSize using the linked streaming example or an equivalent benchmark. Done means the read path uses an evidence-backed chunk size without changing streaming behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 43/100