trailofbits / trailofbits/polyfile
The patched gentoo definition cannot be loaded by libmagic, so magic_defs is no longer a valid magic source
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 390
- Forks
- 31
- Avg merge
- 7h 52m
- Merged PRs (30d)
- 72
Description
Summary
polyfile/magic_defs/gentoo can no longer be loaded by libmagic. The regex at line 38 — PolyFile's
local patch from #3473, which removed a superlinear-backtracking denial-of-service surface — exceeds
what file 5.48 will parse.
$ TZ=UTC ./file/src/file -C -m polyfile/magic_defs/gentoo
polyfile/magic_defs/gentoo, 38: Warning: cannot get string from
`(?<![[:space:]])([[:space:]]+[[:graph:]]([[:print:]]*[[:graph:]])?|[[:space:]][
]* )[[:space:]]+[[:digit:]]+[[:space:]]+[[:'
polyfile/magic_defs/gentoo, 39: Warning: Current entry does not yet have a description for adding a MIME type
The warning text is truncated at the point libmagic gave up, which is the length limit itself.
Why it matters, and why it does not
It does not affect PolyFile users. PolyFile ships its own pure-Python matcher, the rewritten
pattern works correctly there, and tests/test_magic_defs_drift.py passes because gentoo is an
allowlisted local patch mapped to #3473.
It does affect two things:
polyfile/magic_defs/is documented as a hand-maintained copy offile/magic/Magdir/. One entry
is now something upstream libmagic cannot consume, so the directory is no longer a valid magic
source. Anyone copying it into a realfileinstallation gets a broken definition and two
warnings.- Differential tooling has to exclude it. A whole-directory reference run over
magic_defs/
fails ongentoo, and also oncsvandjson— those two are PolyFile-owned definitions for
types 5.48 does not know, so their failure is expected and permanent.gentoois the surprise.
Anyone building a comparison against the reference binary needs all three excluded, and nothing
currently records that.
The real resolution is #3547
The patch exists because PolyFile evaluates regexes with Python's backtracking re where libmagic
uses a POSIX engine. #3547 tracks fixing that at the engine level. If it lands, this patch and the
c-lang one from #3411 can both be reverted, LOCAL_PATCHES empties out, and this problem
disappears rather than being worked around.
Until then, two smaller things would help:
- Record the three unloadable definitions in
docs/updating_libmagic_defs.md, so the next person
building a reference differential does not rediscover it. - Consider whether the rewritten pattern can be expressed within libmagic's length limit without
reintroducing the backtracking. If it can, the patch stays valid in both engines, which is
strictly better.
Found while verifying #3580 against the reference binary.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with polyfile/magic_defs/gentoo and docs/updating_libmagic_defs.md, then reproduce the failure with file/src/file -C -m and review tests/test_magic_defs_drift.py. Done means the unloadable definitions and their differential-tooling implications are recorded, or a libmagic-compatible pattern is demonstrated without restoring the backtracking risk.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100