trailofbits / trailofbits/polyfile

Implement all-offsets scanning for embedded file detection

Open
#3,532 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
390
Forks
31
Avg merge
7h 52m
Merged PRs (30d)
72

Description

Summary

README.md and CLAUDE.md advertise binwalk-style recursive embedded-file detection at every
offset. #3465 removes the dead try_all_offsets parameter that pretended to offer it and corrects
those claims. This issue is where the capability itself would live if anyone wants it.

What exists today

Embedded-file detection comes from three places, none of which scans every offset:

  • format-specific parsers that recurse on their own — polyfile/zipmatcher.py and polyfile/pdf.py
  • polyfile/structmatcher.py:42-44, which writes every bytes field of a parsed struct to a
    temporary file and re-runs the matcher over it
  • individual libmagic definitions that happen to use an offset-agnostic search, such as the
    relaxed ZIP matcher at polyfile/zipmatcher.py:15-20

What this would need

A real scan means running the matcher at every offset in the file, which is quadratic in the input
size before any per-test cost. Three things would have to be settled first:

  1. A cost bound. #3473 and #3527 showed that a single definition can take minutes on a few
    kilobytes. Multiplying the whole matcher by the file length needs a budget, a cap, or both.
  2. Deduplication. A search-based test already matches at any offset, so a naive scan reports
    the same embedded file once per starting position.
  3. A result model. Matcher.match currently yields one match per (mimetype, offset) pair
    after #3466. Thousands of candidate offsets would need either a confidence ranking or a
    containment filter to stay useful.

Filed so the removal in #3465 is a correction rather than a loss of intent. Not scheduled.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading polyfile/structmatcher.py:42-44, polyfile/zipmatcher.py:15-20, and Matcher.match, then review #3465, #3466, #3473, and #3527. The work is done when an offset-wide scan has an explicit cost bound, deduplicates search matches, and provides a useful result model without the described explosion in work or results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.