Unstructured-IO / Unstructured-IO/unstructured

bug/fallback-encoding-detection-for-file-like-objects

Open
#4,434 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
HTML
Stars
15.5k
Forks
1.3k
Avg merge
4d 2h
Merged PRs (30d)
13

Description

Describe the bug
Currently, when trying to determine a file type from its initial text (text_head), the parser attempts fallback character-set detection only for physical file paths. For file-like objects (e.g., SpooledTemporaryFile, BytesIO streams), it blindly decodes the first 4096 bytes using the default encoding (usually UTF-8) with errors="ignore". This can strip characters and lead to corrupted text or incorrect filetype classification when ingesting documents from cloud storage (S3/GCS) or APIs.

To Reproduce
Pass a non-UTF-8 encoded BytesIO object to detect_filetype.

Expected behavior
File-like objects should use the same detect_file_encoding fallback mechanism as physical file paths to avoid data loss on decoding errors.

Screenshots
N/A

Environment Info
N/A - General bug in filetype.py

Additional context
Resolves the TODO in filetype.py:text_head.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in filetype.py at text_head and compare how physical file paths invoke detect_file_encoding with how file-like objects are decoded. Reproduce the issue with a non-UTF-8 BytesIO passed to detect_filetype, then verify that file-like objects use the fallback mechanism without losing characters or misclassifying the file type.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.