trailofbits / trailofbits/polyfile

A level 0 message reached through use or indirect gets a space where libmagic emits a separator

Open
#3,603 1 comment 0 reactions 1 assignee View on GitHub

@ESultanik is already working on this.

Since Sep 18, 2026.

bug
Dominant language
Python
Stars
390
Forks
31
Avg merge
7h 52m
Merged PRs (30d)
72

Description

This supersedes the diagnosis this issue was filed with, and subsumes #3604. The defect is not about \b. PolyFile applies libmagic's continuation spacing rule to every message, including the level 0 messages that a use or an indirect pulls in, where libmagic uses a different rule entirely.

What libmagic does

Two separate rules, in file/src/softmagic.c:

A level 0 entry prints print_sep(ms, *firstline) before its description, never a space (softmagic.c:319-332):

if (*m->desc) {
    *found_match = 1;
    if (print) {
        *returnval = 1;
        *need_separator = 1;
        *printed_something = 1;
        if (print_sep(ms, *firstline) == -1)
            return -1;
        if (mprint(ms, m) == -1)
            return -1;
    }
}

print_sep emits nothing while firstline is set, and FILE_SEPARATOR"\n- " — once it is clear (softmagic.c:2595-2604).

A continuation prints a space, if something printed before it and the entry does not carry NOSPACE (softmagic.c:462-469):

if (*need_separator && (m->flag & NOSPACE) == 0) {
    if (file_printf(ms, " ") == -1)
        return -1;
}

NOSPACE is set when the description begins with \b (apprentice.c:2428-2435).

firstline clears at the end of any match() that printed (softmagic.c:492-500), and the pointer is shared with the recursions that use (softmagic.c:2027) and indirect (softmagic.c:1968-1973) spawn. So a nested run that printed leaves the next level 0 message to be separated by "\n- ".

Evidence

Each line is file 5.48 built from the submodule, against a definition written for the case:

case file PolyFile
use carrying \b baseINNER base INNER
use with no \b baseINNER base INNER
\b on a continuation's own message basenext basenext
one indirect base:first-nested base: first-nested
two indirects base:first-nested\012- second-nested base: first-nested second-nested

The second row is what rules out \b: a use with no \b at all still gets no space. The last row is what shows the separator, and shows it appearing before the second nested verdict rather than the first — the first re-entry printed, which cleared firstline.

That also explains a Mach-O universal binary, where \012- appears three times:

file:     Mach-O universal binary with 2 architectures: [x86_64:\012- Mach-O 64-bit x86_64 executable, flags:<...>] [\012- arm64e (caps: 0x2):\012- Mach-O 64-bit arm64e (caps: PAC00) executable, flags:<...>]
PolyFile: Mach-O universal binary with 2 architectures: [ x86_64: Mach-O 64-bit x86_64 executable, flags:<...>] [ arm64e (caps: 0x2): Mach-O 64-bit arm64e (caps: PAC00) executable, flags:<...>]

The space in : [ is not a separator: magic_defs/cafebabe:21 declares \b [, so the description itself is [.

Where it is

Match._soft_magic_message inserts a space before any message that does not begin with \b, whatever level the test sits at, and never emits MATCH_SEPARATOR within a match.

Each result knows its level through result.test.level, so the level 0 rule is reachable. Modeling firstline needs the recursion boundaries that use and indirect introduce, which the flat result sequence does not record directly, though TestResult.parent does carry the tree.

Why it matters

With #3602 merged, a universal binary's nested descriptions hold the right content and still do not match file byte for byte, for this reason alone. On a 2,000-file sample that is about 40 files, the largest single cluster of strongest-match disagreements.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.