shrink_dataframe skips StringDtype columns with pandas 3
- Dominant language
- Python
- Stars
- 100
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
## Description
`shrink_dataframe()` only applies nullable-integer and low-cardinality category conversions when a column has NumPy `object` dtype. pandas 3 now infers textual columns as `StringDtype` by default, so those documented conversions are skipped.
## Reproduction
With the current unconstrained `pandas` dependency, install pandas 3 and run:
```text
python -m pytest -q -p no:cacheprovider benchmark/test/test_utils.py
```
Two existing tests fail:
```text
TestUtils.test_shrink_dataframe_nullable_int_strict
expected Int8, got StringDtype
TestUtils.test_shrink_dataframe_category
expected CategoricalDtype, got StringDtype
```
The behavior can also be reproduced directly with a `pd.Series(..., dtype="string")`: integer-like values remain strings and repeated low-cardinality values remain `StringDtype`.
## Expected behavior
Text columns should receive the same conservative integer/category handling whether pandas represents them as legacy `object` or `StringDtype`.
## Impact
PXMeter declares `pandas` without an upper bound, so a fresh installation currently selects pandas 3 and leaves the repository test suite failing. Benchmark dataframes also miss the intended memory reductions and dtype normalization.
## Environment
- PXMeter commit: `72117242154d0c9f52ce727dc352210aab76d495`
- Python 3.11.15
- pandas 3.0.5
- Windows 10
Contributor guide
Research direction
Start in benchmark/test/test_utils.py and run python -m pytest -q -p no:cacheprovider benchmark/test/test_utils.py to reproduce the two failures. Trace those tests to shrink_dataframe and compare handling of NumPy object and StringDtype columns. Done means nullable integer and low-cardinality category conversions pass for both representations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100