trailofbits / trailofbits/polyfile
CSV verdicts name the dialect instead of the encoding, and only ASCII and UTF-8 CSV is recognized
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 390
- Forks
- 31
- Avg merge
- 7h 52m
- Merged PRs (30d)
- 72
Description
Summary
file_is_csv prints CSV <encoding> text and parses the file's raw bytes, so file names the
text encoding in every CSV verdict and accepts CSV in any encoding whose commas and newlines
survive as bytes. PolyFile's CSVTest and polyfile/magic_defs/csv do neither: the message names
the Python dialect instead of the encoding, and the test decodes the buffer as UTF-8 first, so
every CSV that is not ASCII or UTF-8 is missed.
Found while fixing #3537, which asked whether CSV carries the byte-order-mark divergence #3500
recorded for JSON. It does not: a BOM-prefixed UTF-8 CSV is CSV to file and to PolyFile alike,
because csv_parse walks bytes and the mark only joins the first field. These two divergences are
what is actually there.
Reproducer
$ printf 'a,b,c\n1,2,3\n4,5,6\n' > plain.csv
$ printf '\xef\xbb\xbfa,b,c\n1,2,3\n4,5,6\n' > bom.csv
$ printf 'a,b,caf\xe9\n1,2,3\n4,5,6\n' > latin1.csv
$ python3 -c 'open("utf16.csv","wb").write("a,b,c\n1,2,3\n4,5,6\n".encode("utf-16-le"))'
| file | file -b |
PolyFile |
|---|---|---|
plain.csv |
CSV ASCII text |
CSV text (excel dialect) |
bom.csv |
CSV Unicode text, UTF-8 (with BOM) text |
CSV text (excel dialect) |
latin1.csv |
CSV ISO-8859 text |
ISO-8859 text |
utf16.csv |
CSV Unicode text, UTF-16, little-endian text |
Unicode text, UTF-16, little-endian text |
Defect 1: the message names the dialect, not the encoding
file/src/is_csv.c:157-158:
if (file_printf(ms, "CSV %s%stext", code ? code : "",
code ? " " : "") == -1)
code is the file_encoding verdict that file_buffer computed at file/src/funcs.c:369, so the
encoding name belongs to the CSV message itself rather than to a description appended after it.
polyfile/magic_defs/csv:6 prints CSV text (%s dialect) instead, where %s is whichever name
from csv.list_dialects() accepted the buffer. That name has no counterpart in file, and the
encoding it replaced has no other way into the verdict, because CSVTest.precedes_soft_magic ends
the run before the text description is appended.
After #3537 the string this needs for a marked file is available:
LIBMAGIC_ENCODING_NAMES["utf-8-sig"].
Defect 2: only ASCII and UTF-8 CSV is recognized
CSVTest.test (polyfile/magic.py:3865) starts with
text = data[absolute_offset:].decode("utf-8")
and returns a FailedTest on UnicodeDecodeError, so Latin-1, UTF-16 and UTF-32 CSV never reach
the dialect loop. csv_parse counts ", , and \n bytes and ignores everything else
(file/src/is_csv.c:129-190), so the NUL-interleaved commas and newlines of UTF-16 parse as
ordinary field and row separators and a high byte in Latin-1 is just another field character.
Scope
Pre-existing, and not a regression from anything in v0.6.0. No corpus stem in file/tests/ is a
CSV, so test_file_corpus does not cover either half.
Defect 1 changes the message of every CSV match, so it is worth deciding deliberately whether
PolyFile keeps the dialect name it reports today. Defect 2 only adds matches.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CSVTest.test in polyfile/magic.py and the definition in polyfile/magic_defs/csv, then compare their behavior with the supplied plain, BOM-prefixed, Latin-1, and UTF-16 reproducers. Review the referenced file/src/is_csv.c behavior and the available LIBMAGIC_ENCODING_NAMES entry. Done means CSV verdicts report the encoding consistently and non-UTF-8 CSV inputs are recognized, with the dialect-label decision resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100