scikit-learn / scikit-learn/scikit-learn

BUG: Silent float32 upcasting to float64 in Pipeline/StandardScaler causes 2x memory bloat

Open
#34,032 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug Needs Investigation
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

[!WARNING]
This issue is not yet ready for a PR. If you are interested in contributing to scikit-learn, please have a look at our contributing guidelines, and in particular the sections for new contributors and the "Needs triage" label.

Describe the bug and give evidence about its user-facing impact

When processing massive financial tick data (e.g., shape (10_000_000, 50)), quant systems typically cast down to np.float32 to keep the data within RAM limits.

However, passing a float32 NumPy array through a Pipeline containing a StandardScaler silently upcasts the transformed output back to float64. This creates a silent memory copy that doubles the RAM requirement, frequently triggering Out-Of-Memory (OOM) kills on high-frequency streaming applications.

Steps/Code to Reproduce
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Generate a mock financial tick dataset in float32
X_financial_ticks = np.random.randn(1_000_000, 10).astype(np.float32)

print(f"Input dtype: {X_financial_ticks.dtype}")
print(f"Input memory: {X_financial_ticks.nbytes / 1e6} MB")

# Pass through a standard pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler())
])

X_transformed = pipeline.fit_transform(X_financial_ticks)

print(f"Output dtype: {X_transformed.dtype}")
print(f"Output memory: {X_transformed.nbytes / 1e6} MB")
Expected Results

Expected Results:

Input dtype: float32
Input memory: 40.0 MB
Output dtype: float32
Output memory: 40.0 MB
Actual Results

Actual Results:

Input dtype: float32
Input memory: 40.0 MB
Output dtype: float64
Output memory: 80.0 MB  <-- 100% memory bloat
Versions
System:
    python: 3.13.11 (main, Dec  8 2025, 11:43:54) [GCC 15.2.0]
executable: /usr/bin/python
   machine: Linux-6.17.10+kali-amd64-x86_64-with-glibc2.42

Python dependencies:
      sklearn: 1.8.0
          pip: 25.3
   setuptools: 78.1.1
        numpy: 2.3.5
        scipy: 1.16.3
       Cython: None
       pandas: 2.3.3
   matplotlib: 3.10.7+dfsg1
       joblib: 1.5.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 4
         prefix: libgomp
       filepath: /home/killer/.local/lib/python3.13/site-packages/scikit_learn.libs/libgomp-e985bcbb.so.1.0.0
        version: None
Interest in fixing the bug

Proposed Solution:
I would like to investigate the check_array / dtype handling inside StandardScaler.transform to ensure it respects the input dtype without silently upcasting, provided the user explicitly inputs float32.
If the maintainers agree this is an anti-pattern, please assign this to me. I am happy to open a PR to patch the data-copying behavior.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with StandardScaler.transform and its check_array/dtype handling, using the provided float32 Pipeline reproduction to inspect the output dtype and memory use. Done means the expected float32 output is preserved without the reported upcast, with regression coverage for this behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.