trailofbits / trailofbits/polyfile

CSV verdicts name the dialect instead of the encoding, and only ASCII and UTF-8 CSV is recognized

Open
#3,554 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
390
Forks
31
Avg merge
7h 52m
Merged PRs (30d)
72

Description

Summary

file_is_csv prints CSV <encoding> text and parses the file's raw bytes, so file names the
text encoding in every CSV verdict and accepts CSV in any encoding whose commas and newlines
survive as bytes. PolyFile's CSVTest and polyfile/magic_defs/csv do neither: the message names
the Python dialect instead of the encoding, and the test decodes the buffer as UTF-8 first, so
every CSV that is not ASCII or UTF-8 is missed.

Found while fixing #3537, which asked whether CSV carries the byte-order-mark divergence #3500
recorded for JSON. It does not: a BOM-prefixed UTF-8 CSV is CSV to file and to PolyFile alike,
because csv_parse walks bytes and the mark only joins the first field. These two divergences are
what is actually there.

Reproducer

$ printf 'a,b,c\n1,2,3\n4,5,6\n' > plain.csv
$ printf '\xef\xbb\xbfa,b,c\n1,2,3\n4,5,6\n' > bom.csv
$ printf 'a,b,caf\xe9\n1,2,3\n4,5,6\n' > latin1.csv
$ python3 -c 'open("utf16.csv","wb").write("a,b,c\n1,2,3\n4,5,6\n".encode("utf-16-le"))'
file file -b PolyFile
plain.csv CSV ASCII text CSV text (excel dialect)
bom.csv CSV Unicode text, UTF-8 (with BOM) text CSV text (excel dialect)
latin1.csv CSV ISO-8859 text ISO-8859 text
utf16.csv CSV Unicode text, UTF-16, little-endian text Unicode text, UTF-16, little-endian text

Defect 1: the message names the dialect, not the encoding

file/src/is_csv.c:157-158:

	if (file_printf(ms, "CSV %s%stext", code ? code : "",
	    code ? " " : "") == -1)

code is the file_encoding verdict that file_buffer computed at file/src/funcs.c:369, so the
encoding name belongs to the CSV message itself rather than to a description appended after it.
polyfile/magic_defs/csv:6 prints CSV text (%s dialect) instead, where %s is whichever name
from csv.list_dialects() accepted the buffer. That name has no counterpart in file, and the
encoding it replaced has no other way into the verdict, because CSVTest.precedes_soft_magic ends
the run before the text description is appended.

After #3537 the string this needs for a marked file is available:
LIBMAGIC_ENCODING_NAMES["utf-8-sig"].

Defect 2: only ASCII and UTF-8 CSV is recognized

CSVTest.test (polyfile/magic.py:3865) starts with

text = data[absolute_offset:].decode("utf-8")

and returns a FailedTest on UnicodeDecodeError, so Latin-1, UTF-16 and UTF-32 CSV never reach
the dialect loop. csv_parse counts ", , and \n bytes and ignores everything else
(file/src/is_csv.c:129-190), so the NUL-interleaved commas and newlines of UTF-16 parse as
ordinary field and row separators and a high byte in Latin-1 is just another field character.

Scope

Pre-existing, and not a regression from anything in v0.6.0. No corpus stem in file/tests/ is a
CSV, so test_file_corpus does not cover either half.

Defect 1 changes the message of every CSV match, so it is worth deciding deliberately whether
PolyFile keeps the dialect name it reports today. Defect 2 only adds matches.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CSVTest.test in polyfile/magic.py and the definition in polyfile/magic_defs/csv, then compare their behavior with the supplied plain, BOM-prefixed, Latin-1, and UTF-16 reproducers. Review the referenced file/src/is_csv.c behavior and the available LIBMAGIC_ENCODING_NAMES entry. Done means CSV verdicts report the encoding consistently and non-UTF-8 CSV inputs are recognized, with the dialect-label decision resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.