haskell-servant / haskell-servant/servant

SourceT's readFile uses an innefficient chunk size

Open
#1,770 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
2k
Forks
427
Avg merge
2d 23h
Merged PRs (30d)
5

Description

4096 is not the most optimal chunk size, and so this function is slower at reading files than it could be.

In a quick benchmark, using the example from https://docs.servant.dev/en/stable/cookbook/basic-streaming/Streaming.html, and a 1 gb "README.md" file, curl to /dev/null takes 1.84 seconds.

I was able to optimize that to 0.68 seconds. My optimisation happened to read it into a lazy ByteString and then stream out the toChunks, which is ugly and I don't recommend. (I was actually just investigating how to stream a lazy ByteString through servant when I noticed this performance problem.)

Probably the right fix is to change 4096 to some more optimal chunk size. Data.ByteString.Lazy.Internal.defaultChunkSize is a good one.

Contributor guide

Open the contributing guide

Research direction

Locate SourceT's readFile implementation and inspect how it chooses the 4096-byte chunk size. Compare the current behavior with Data.ByteString.Lazy.Internal.defaultChunkSize using the linked streaming example or an equivalent benchmark. Done means the read path uses an evidence-backed chunk size without changing streaming behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
43/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.