Unstructured-IO / Unstructured-IO/unstructured
partition_csv() raises UnicodeDecodeError on UTF-16-LE files even with correct encoding argument
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 15.5k
- Forks
- 1.3k
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 13
Description
Describe the bug
Delimiter sniffing in partition_csv() decodes its sample line-by-line (unstructured/partition/csv.py, _CsvPartitioningContext.delimiter). readlines() splits the raw bytes on 0x0A, which in UTF-16 lands in the middle of a code unit, and only the first fragment carries the BOM. For UTF-16-LE — what str.encode("utf-16") and Excel's "Unicode Text" export produce — the first fragment is odd-length and decoding raises UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x0a ... truncated data, even though the caller passed the correct encoding and pd.read_csv handles the same bytes fine.
UTF-16-BE doesn't raise, but silently decodes every line after the first as mojibake, so the sniffer sees corrupted data. (The existing stanley-cups-utf-16.csv example doc is BE, which is why tests never hit this.)
To Reproduce
import io
from unstructured.partition.csv import partition_csv
data = "name,age\nAlice,30\nBob,25\n".encode("utf-16")
partition_csv(file=io.BytesIO(data), encoding="utf-16")
# UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x0a in position 18: truncated data
Fails identically with filename=. pd.read_csv(io.BytesIO(data), encoding="utf-16") parses the same bytes correctly.
Expected behavior
The file partitions successfully when the correct encoding is supplied (and delimiter sniffing sees correctly decoded text for both UTF-16 variants).
Environment
unstructured 0.26.3, Python 3.11, reproduced on current main.
I have a fix ready and will open a PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in unstructured/partition/csv.py at _CsvPartitioningContext.delimiter and reproduce the UTF-16-LE example with partition_csv. Check the delimiter-sniffing path against pd.read_csv behavior for both UTF-16 variants; done means correct decoding and successful partitioning when encoding is supplied, with regression coverage for the reported failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100