trailofbits / trailofbits/polyfile
No equivalent of libmagic's readcdf, so an HWP 5.0 document keeps the OLE 2 prefix
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 390
- Forks
- 31
- Avg merge
- 7h 52m
- Merged PRs (30d)
- 72
Description
Summary
libmagic has a built-in compound-document reader, readcdf.c, that runs ahead of soft magic and
reports some OLE 2 payloads on its own. PolyFile has no equivalent, so for those payloads it can
only report the libmagic definition that matched, which nests the description under the OLE 2
header entry. The one corpus file that exercises this is an HWP 5.0 document: file describes it
as Hancom HWP (Hangul Word Processor) file, version 5.0, and PolyFile describes it as
OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0.
Reproducer
$ TZ=UTC ./file/src/file -b -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
Hancom HWP (Hangul Word Processor) file, version 5.0
$ TZ=UTC ./file/src/file -b --mime-type -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
application/x-hwp
$ python -c '
import polyfile.magic
from polyfile.magic import MagicMatcher
polyfile.magic.local_date = polyfile.magic.utc_date
with open("file/tests/HWP2016.hwp.testfile", "rb") as f:
for match in MagicMatcher.DEFAULT_INSTANCE.match(f.read()):
print(str(match))
'
OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0
$ polyfile --only-match-mime file/tests/HWP2016.hwp.testfile
application/hwp+zip
So both the description and the MIME type diverge.
Cause
This is not a definitions-handling bug. The standalone description does not come from the magic
definitions at all. file -d shows it arriving before soft magic runs:
$ TZ=UTC ./file/src/file -d -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
[try cdf 1]
file/tests/HWP2016.hwp.testfile: Hancom HWP (Hangul Word Processor) file, version 5.0
file_buffer calls file_trycdf before the soft-magic pass, and a hit ends the run unless
MAGIC_CONTINUE is set (file/src/funcs.c:443-452). file_trycdf parses the compound document,
reads its FileHeader user stream, compares the first bytes against the HWP Document File
signature, and prints the description itself (file/src/readcdf.c:633-646):
if (cdf_read_user_stream(&info, &h, &sat, &ssat, &sst, &dir,
"FileHeader", &scn) != -1) {
#define HWP5_SIGNATURE "HWP Document File"
if (scn.sst_len * scn.sst_ss >= sizeof(HWP5_SIGNATURE) - 1
&& memcmp(scn.sst_tab, HWP5_SIGNATURE,
sizeof(HWP5_SIGNATURE) - 1) == 0) {
if (NOTMIME(ms)) {
if (file_printf(ms,
"Hancom HWP (Hangul Word Processor) file, version 5.0") == -1)
The definitions' only version 5.0 entry is polyfile/magic_defs/ole2compounddocs:269. It matches
the directory entry name FileHeader as a lestring16 relative to the OLE 2 header, so it carries
the : continuation prefix and the OLE 2 description ahead of it, and it declares
!:mime application/hwp+zip where readcdf.c emits application/x-hwp. PolyFile reports exactly
what the definitions say; what it lacks is the built-in reader that short-circuits them.
Running file with -k shows all three matches, and PolyFile reports the middle one:
$ TZ=UTC ./file/src/file -b -k -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
Hancom HWP (Hangul Word Processor) file, version 5.0\012- OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0\012- data
Consequences
tests/test_magic.py:corpus_result_matches carries an escape hatch for this one stem: it accepts a
suffix match instead of an exact one, because the expected description is the tail of what PolyFile
reports. #3501 documents that branch and points here for the cause; the branch has to stay until
PolyFile can read compound documents.
HWP2016.hwp is the only compound document in the libmagic corpus, so the corpus does not measure
how much else readcdf.c affects. Its other outputs include the summary-information fields
(CDF V2 Document, ...), the CLSID line, and the Microsoft ... verdicts it derives from the
document's own streams.
Scope
libmagic's reader is file/src/cdf.c (1682 lines), file/src/cdf.h (363), and
file/src/readcdf.c (702) — a subsystem, not a definition fix, so this needs a decision about
whether PolyFile wants a compound-document reader at all. The narrow version, enough for this
divergence, is a reader that walks the directory, extracts the FileHeader stream, and matches its
signature; the full version reproduces the summary-information parsing too.
Precedent: #3488 added the equivalent of file_ascmagic for the same class of gap.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with polyfile/magic_defs/ole2compounddocs and the matching flow in polyfile.magic, then compare file/src/cdf.c, file/src/cdf.h, and file/src/readcdf.c. Run the HWP2016 reproducer and tests/test_magic.py:corpus_result_matches; done means the compound-document case is handled without the suffix escape hatch and its description and MIME match the referenced file output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- reverse-engineering, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100