apache / apache/hamilton

Deprecation warnings in Pandas CSV Reader

Open
#1,417 1 comment 0 reactions 1 assignee Claimed by @iprithv View on GitHub
i/o python
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

Deprecation warnings in Pandas CSV Reader (`pandas_extensions.py`) due to deprecated pandas arguments.

# Current behavior

Running the Hamilton test suite on the latest `main` branch generates multiple `FutureWarning` messages from `pandas.read_csv`, triggered by deprecated keyword arguments in `PandasCSVReader`:

- `keep_date_col`
- `verbose`
- `delim_whitespace`

These warnings originate from:
`hamilton/plugins/pandas_extensions.py` inside `PandasCSVReader._get_loading_kwargs()`.

## Stack Traces
Example warnings:

FutureWarning: The 'keep_date_col' keyword in pd.read_csv is deprecated and will be removed in a future version.
FutureWarning: The 'delim_whitespace' keyword in pd.read_csv is deprecated and will be removed in a future version. Use sep='\s+' instead.
FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.

## Steps to replicate behavior
1. Run: `pytest -q`
2. Observe pandas `FutureWarning` output from CSV-related tests

## Library & System Information
- Hamilton: latest `main`
- pandas: 2.2.x
- Python: 3.11
- macOS

# Expected behavior

Hamilton should avoid passing deprecated arguments to `pandas.read_csv`, preventing warnings and ensuring compatibility with future pandas versions.

Specifically:
- Remove deprecated args (`keep_date_col`, `verbose`)
- Replace `delim_whitespace=True` with recommended `sep=r"\s+"`
- Ensure test suite runs with zero deprecation warnings from pandas

# Additional context

Fixing this will:
- Clean up test output
- Improve compatibility with pandas ≥ 2.0
- Prevent potential future runtime errors when pandas removes these parameters

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.