trailofbits / trailofbits/polyfile

The patched gentoo definition cannot be loaded by libmagic, so magic_defs is no longer a valid magic source

Open
#3,586 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
390
Forks
31
Avg merge
7h 52m
Merged PRs (30d)
72

Description

Summary

polyfile/magic_defs/gentoo can no longer be loaded by libmagic. The regex at line 38 — PolyFile's
local patch from #3473, which removed a superlinear-backtracking denial-of-service surface — exceeds
what file 5.48 will parse.

$ TZ=UTC ./file/src/file -C -m polyfile/magic_defs/gentoo
polyfile/magic_defs/gentoo, 38: Warning: cannot get string from
  `(?<![[:space:]])([[:space:]]+[[:graph:]]([[:print:]]*[[:graph:]])?|[[:space:]][	
]* )[[:space:]]+[[:digit:]]+[[:space:]]+[[:'
polyfile/magic_defs/gentoo, 39: Warning: Current entry does not yet have a description for adding a MIME type

The warning text is truncated at the point libmagic gave up, which is the length limit itself.

Why it matters, and why it does not

It does not affect PolyFile users. PolyFile ships its own pure-Python matcher, the rewritten
pattern works correctly there, and tests/test_magic_defs_drift.py passes because gentoo is an
allowlisted local patch mapped to #3473.

It does affect two things:

  1. polyfile/magic_defs/ is documented as a hand-maintained copy of file/magic/Magdir/. One entry
    is now something upstream libmagic cannot consume, so the directory is no longer a valid magic
    source. Anyone copying it into a real file installation gets a broken definition and two
    warnings.
  2. Differential tooling has to exclude it. A whole-directory reference run over magic_defs/
    fails on gentoo, and also on csv and json — those two are PolyFile-owned definitions for
    types 5.48 does not know, so their failure is expected and permanent. gentoo is the surprise.
    Anyone building a comparison against the reference binary needs all three excluded, and nothing
    currently records that.

The real resolution is #3547

The patch exists because PolyFile evaluates regexes with Python's backtracking re where libmagic
uses a POSIX engine. #3547 tracks fixing that at the engine level. If it lands, this patch and the
c-lang one from #3411 can both be reverted
, LOCAL_PATCHES empties out, and this problem
disappears rather than being worked around.

Until then, two smaller things would help:

  • Record the three unloadable definitions in docs/updating_libmagic_defs.md, so the next person
    building a reference differential does not rediscover it.
  • Consider whether the rewritten pattern can be expressed within libmagic's length limit without
    reintroducing the backtracking. If it can, the patch stays valid in both engines, which is
    strictly better.

Found while verifying #3580 against the reference binary.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with polyfile/magic_defs/gentoo and docs/updating_libmagic_defs.md, then reproduce the failure with file/src/file -C -m and review tests/test_magic_defs_drift.py. Done means the unloadable definitions and their differential-tooling implications are recorded, or a libmagic-compatible pattern is demonstrated without restoring the backtracking risk.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.