Potential encoder speedup on non-win32
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 2.4k
- Forks
- 363
- PR merge metrics
- No merged PRs in 30d
Description
This ( https://hydrogenaud.io/index.php/topic,123889.msg1024853.html#msg1024853 ) post prompted me to poke around fwrite and hack in setting the fwrite output buffer size for non-win32 targets: https://github.com/chocolate42/flac/tree/vbuf
Tested by re-encoding a ~150MB 24 bit flac file to -0 to the same drive, on a 2.5" HDD and a slowish SSD on a skylake laptop. Tried a buffer of 10MB to match the win32 path, and a buffer of 10MB+8 based on the recommendations in this post: https://www.enterprisestorageforum.com/hardware/a-trip-down-the-data-path-i-o-and-performance/
These are the average wall times of 10 runs each in seconds on Linux 64 bit:
HDD SSD
default 15.156 7.564
10mb 10.081 6.799
10mb+8 9.904 6.752
- This shows the best case of -0, as compression level increases the relevance of I/O performance reduces
- I had to provide a buffer to setvbuf, apparently when NULL is provided the implementation is free to ignore the request to increase the internal buffer and glibc does ignore it.
- The default on my machine is either 4096 or 8192 (or 8200 if the +8 is important)
- Potentially something similar could be done for reading, haven't looked into it at all yet
- Similarly benefits stdout with -c too
- Haven't (and can't) tested win32 path to confirm if it is actually using a 10MB buffer. It probably is but unconfirmed
Didn't make a proper PR directly because there's some open questions
- Only tried on Linux 64 bit
- Does it work with stdout always? Is there a weird architecture or something an external program could do (like redefining stdout) that might make it fail? Should we care even if an oddball config might fail?
- Embedded can't malloc a large buffer, embedded should probably be excluded
- Is setvbuf even available on embedded?
- Does it matter if setvbuf fails? Surely it'll just revert to the default instead of leaving the stream in some bad state
- What's the deal with the win32 path checking that it's a "real" file? /dev/null seems to work. Could the paths be unified?
- Is the +8 actually beneficial or is it just a case of something that used to be true
- Haven't tried buffer sizes other than ~10MB
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source file or test. Start by tracing the non-win32 encoder's fwrite path and comparing it with the win32 path, then investigate setvbuf behavior for stdout and embedded targets. Done means the supported scope, buffer size, failure behavior, and performance validation are agreed and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- audio-video-rtc, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100