trailofbits / trailofbits/polyfile
Implement all-offsets scanning for embedded file detection
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 390
- Forks
- 31
- Avg merge
- 7h 52m
- Merged PRs (30d)
- 72
Description
Summary
README.md and CLAUDE.md advertise binwalk-style recursive embedded-file detection at every
offset. #3465 removes the dead try_all_offsets parameter that pretended to offer it and corrects
those claims. This issue is where the capability itself would live if anyone wants it.
What exists today
Embedded-file detection comes from three places, none of which scans every offset:
- format-specific parsers that recurse on their own —
polyfile/zipmatcher.pyandpolyfile/pdf.py polyfile/structmatcher.py:42-44, which writes everybytesfield of a parsed struct to a
temporary file and re-runs the matcher over it- individual libmagic definitions that happen to use an offset-agnostic
search, such as the
relaxed ZIP matcher atpolyfile/zipmatcher.py:15-20
What this would need
A real scan means running the matcher at every offset in the file, which is quadratic in the input
size before any per-test cost. Three things would have to be settled first:
- A cost bound. #3473 and #3527 showed that a single definition can take minutes on a few
kilobytes. Multiplying the whole matcher by the file length needs a budget, a cap, or both. - Deduplication. A
search-based test already matches at any offset, so a naive scan reports
the same embedded file once per starting position. - A result model.
Matcher.matchcurrently yields one match per(mimetype, offset)pair
after #3466. Thousands of candidate offsets would need either a confidence ranking or a
containment filter to stay useful.
Filed so the removal in #3465 is a correction rather than a loss of intent. Not scheduled.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading polyfile/structmatcher.py:42-44, polyfile/zipmatcher.py:15-20, and Matcher.match, then review #3465, #3466, #3473, and #3527. The work is done when an offset-wide scan has an explicit cost bound, deduplicates search matches, and provides a useful result model without the described explosion in work or results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100