jam1garner / jam1garner/binrw

Unnecessary rewinds and re-reads of magic on data enums cause bad performance and prevent use of `NoSeek`

Open
#253 9 comments 0 reactions 0 assignees View on GitHub
bug performance issue
Dominant language
Rust
Stars
853
Forks
56
PR merge metrics
No merged PRs in 30d

Description

I'm using binrw to parse a format with an enum with around 300 variants(all using magic directives) and with files that can contain up to 40 000 items. The performance when parsing files weren't what I expected, even if I add `return_unexpected_error`, although that did help. I did some profiling and saw that there's a lot of allocations happening as part of the magic parsing due to the boxed value in the generated error. As an experiment I tried replacing the `BadMagic` error returned by the `magic` function with `NoVariantMatch`(which only contains the position) and saw a +70% decrease in parse time. Considering that the value isn't even used when `return_unexpected_error` is enabled this seems like a good opportunity for optimization. It may be possible to improve for `return_all_errors` as well since the possible data types for `magic` are quite small.

I've created a [repo](https://github.com/johanholmerin/binrw-slow-enum-magic) with a simple reproduction. If you replace `BadMagic` with `NoVariantMatch` in binrw you should see the difference in performance.

I'm would also be interested if there's any performance to be gained by not re-reading the magic value for each variant. In my case I'm parsing from in-memory, but there's a lot of reading and seeking being done that's unnecessary. It would require a larger change so I've not done any testing but might be worth looking into.

Contributor guide

Open the contributing guide

Research direction

Start with the magic function and the BadMagic and NoVariantMatch errors, then reproduce the allocations and parsing time using the linked binrw-slow-enum-magic repository. Check behavior with return_unexpected_error and return_all_errors, and investigate the repeated reads and seeks relevant to NoSeek. Done means the reproduction improves without breaking the existing error behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
devtools
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.