bytedance / bytedance/PXMeter

shrink_dataframe skips StringDtype columns with pandas 3

Open Beginner friendly
#13 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
100
Forks
12
PR merge metrics
No merged PRs in 30d

Description

## Description

`shrink_dataframe()` only applies nullable-integer and low-cardinality category conversions when a column has NumPy `object` dtype. pandas 3 now infers textual columns as `StringDtype` by default, so those documented conversions are skipped.

## Reproduction

With the current unconstrained `pandas` dependency, install pandas 3 and run:

```text
python -m pytest -q -p no:cacheprovider benchmark/test/test_utils.py
```

Two existing tests fail:

```text
TestUtils.test_shrink_dataframe_nullable_int_strict
expected Int8, got StringDtype

TestUtils.test_shrink_dataframe_category
expected CategoricalDtype, got StringDtype
```

The behavior can also be reproduced directly with a `pd.Series(..., dtype="string")`: integer-like values remain strings and repeated low-cardinality values remain `StringDtype`.

## Expected behavior

Text columns should receive the same conservative integer/category handling whether pandas represents them as legacy `object` or `StringDtype`.

## Impact

PXMeter declares `pandas` without an upper bound, so a fresh installation currently selects pandas 3 and leaves the repository test suite failing. Benchmark dataframes also miss the intended memory reductions and dtype normalization.

## Environment

- PXMeter commit: `72117242154d0c9f52ce727dc352210aab76d495`
- Python 3.11.15
- pandas 3.0.5
- Windows 10

Contributor guide

Open the contributing guide

Research direction

Start in benchmark/test/test_utils.py and run python -m pytest -q -p no:cacheprovider benchmark/test/test_utils.py to reproduce the two failures. Trace those tests to shrink_dataframe and compare handling of NumPy object and StringDtype columns. Done means nullable integer and low-cardinality category conversions pass for both representations.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.