CSVDocumentCleaner silently removes nonempty NA and NULL strings

Describe the bug

CSVDocumentCleaner converts nonempty strings such as NA, NULL, N/A, NaN, and None into missing values while reading CSV content. This can delete entire rows/columns or silently replace cells with empty strings. It also happens with both cleaning options disabled.

For example, NA can be a region/category code that should survive indexing into a RAG knowledge base.

Reproduce

import pandas  # Initialize the optional CSV dependency before Haystack's lazy imports.
from haystack import Document
from haystack.components.preprocessors.csv_document_cleaner import CSVDocumentCleaner

csv = "region,value\nNA,NULL\nN/A,NaN\nNone,null\n"
result = CSVDocumentCleaner().run(documents=[Document(content=csv)])
print(repr(result["documents"][0].content))
# Actual: 'region,value\n'
# Expected: the original nonempty rows are preserved.

csv = "NA,NULL\n,\n"
cleaner = CSVDocumentCleaner(remove_empty_rows=False, remove_empty_columns=False)
result = cleaner.run(documents=[Document(content=csv)])
print(repr(result["documents"][0].content))
# Actual: ',\n,\n'
# Expected: 'NA,NULL\n,\n'

Cause and proposed fix

The pd.read_csv(..., header=None, dtype=object) call keeps pandas' default NA markers. dtype=object does not prevent this conversion.

Using keep_default_na=False, na_values=[""] preserves nonempty literal strings while retaining missing-value detection for genuinely empty fields. This behavior is documented in pandas.read_csv.

A local patch includes three regression tests covering literal values, empty rows/columns, and disabled cleaning. All three fail against the unchanged source; the patch passes all 21 tests in the CSVDocumentCleaner module, with 100% statement coverage for that module. Ruff and focused mypy checks pass. This changes behavior for applications that intentionally use textual NA markers; those applications would need to normalize the markers to empty fields first.

Environment

Windows, Python 3.12.14, Haystack source 3.2.0rc0 (main observed at 9c7e2a9c7f96930ed05d3de8a3bc770f56f777fc), pandas 2.2.3, NumPy 2.2.6.

Tests ran through Hatch in a focused environment, with pandas/NumPy preloaded to avoid a separate optional-import reload problem in that environment. No live LLM service was used.

AI assistance

This report, investigation, local patch, and tests were prepared and executed with Codex. No human review is claimed. The fix and regression tests are in #12772.