crewAIInc / crewAIInc/crewAI

[BUG] CSVLoader misparses headers in UTF-8 CSV files with a BOM

Open Beginner friendly
#7,466 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

CSVLoader passes a leading UTF-8 BOM (U+FEFF) to csv.DictReader. For an ordinary header, the BOM becomes part of the first column name. When the first header is quoted and contains a comma, it also prevents quote recognition: a two-column CSV is parsed as three columns and row values are assigned to the wrong headers.

Reproduced on main at 66ef97c73e5459059274c5117e48e2e5ba50a871 with both inline text and a local UTF-8-with-BOM file. No LLM or API key is needed.

Steps to Reproduce

Run this against the current workspace:

from crewai_tools.rag.loaders.csv_loader import CSVLoader
from crewai_tools.rag.source_content import SourceContent

csv_text = '\ufeff"last, first",age\n"Doe, Jane",30\n'
result = CSVLoader().load(SourceContent(csv_text))
print(repr(result.metadata["columns"]))
print(result.content)

The same result occurs when writing this text to a file with UTF-8 encoding and loading that path.

Expected behavior

The columns should be ["last, first", "age"], and the row should contain last, first: Doe, Jane | age: 30, just as it does without the leading BOM.

Screenshots/Code snippets

Actual columns:

['\ufeff"last', ' first"', 'age']

The value 30 is assigned to the spurious first" header rather than age. An unquoted name,age header instead produces ["\ufeffname", "age"].

Operating System

Other (specify in additional context)

Python Version

3.12

crewAI Version

1.15.21 (main @ 66ef97c)

crewAI Tools Version

1.15.21 (main @ 66ef97c)

Virtual Environment

Venv

Evidence

In lib/crewai-tools/src/crewai_tools/rag/loaders/csv_loader.py, _load_from_file decodes using UTF-8 and _parse_csv directly passes StringIO(content) to csv.DictReader. The leading U+FEFF is therefore preserved before the opening quote. Removing only that prefix makes the same input parse correctly.

Possible Solution

Remove one leading U+FEFF in _parse_csv before creating csv.DictReader, so file, inline-text and URL inputs share the same normalization. Add regression coverage for quoted/unquoted headers with and without a BOM, and preserve U+FEFF inside field values.

Additional context

AI-assisted contribution: this report and the accompanying fix were prepared with Codex. Per CONTRIBUTING.md, please apply the llm-generated label; the issue form does not expose a label editor for this account.

Duplicate check: searched this repository for CSVLoader, BOM and utf-8-sig, and checked the open PR titles and descriptions. No matching CSVLoader fix was found.

Local reproduction: macOS 26.6.2, Python 3.12.13. The regression suite also reproduces the issue on Python 3.13.13: 7 BOM cases fail before the fix, while 16 cases pass.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/crewai-tools/src/crewai_tools/rag/loaders/csv_loader.py, reading _load_from_file and _parse_csv to trace how inline, file, and URL content reaches csv.DictReader. Use the reported BOM examples as regression cases, covering quoted and unquoted headers while preserving U+FEFF inside field values. Done means BOM-prefixed input produces the same columns and row content as input without the BOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.