scikit-learn / scikit-learn/scikit-learn
BUG: Silent float32 upcasting to float64 in Pipeline/StandardScaler causes 2x memory bloat
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
[!WARNING]
This issue is not yet ready for a PR. If you are interested in contributing to scikit-learn, please have a look at our contributing guidelines, and in particular the sections for new contributors and the "Needs triage" label.
Describe the bug and give evidence about its user-facing impact
When processing massive financial tick data (e.g., shape (10_000_000, 50)), quant systems typically cast down to np.float32 to keep the data within RAM limits.
However, passing a float32 NumPy array through a Pipeline containing a StandardScaler silently upcasts the transformed output back to float64. This creates a silent memory copy that doubles the RAM requirement, frequently triggering Out-Of-Memory (OOM) kills on high-frequency streaming applications.
Steps/Code to Reproduce
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# Generate a mock financial tick dataset in float32
X_financial_ticks = np.random.randn(1_000_000, 10).astype(np.float32)
print(f"Input dtype: {X_financial_ticks.dtype}")
print(f"Input memory: {X_financial_ticks.nbytes / 1e6} MB")
# Pass through a standard pipeline
pipeline = Pipeline([
('scaler', StandardScaler())
])
X_transformed = pipeline.fit_transform(X_financial_ticks)
print(f"Output dtype: {X_transformed.dtype}")
print(f"Output memory: {X_transformed.nbytes / 1e6} MB")
Expected Results
Expected Results:
Input dtype: float32
Input memory: 40.0 MB
Output dtype: float32
Output memory: 40.0 MB
Actual Results
Actual Results:
Input dtype: float32
Input memory: 40.0 MB
Output dtype: float64
Output memory: 80.0 MB <-- 100% memory bloat
Versions
System:
python: 3.13.11 (main, Dec 8 2025, 11:43:54) [GCC 15.2.0]
executable: /usr/bin/python
machine: Linux-6.17.10+kali-amd64-x86_64-with-glibc2.42
Python dependencies:
sklearn: 1.8.0
pip: 25.3
setuptools: 78.1.1
numpy: 2.3.5
scipy: 1.16.3
Cython: None
pandas: 2.3.3
matplotlib: 3.10.7+dfsg1
joblib: 1.5.2
threadpoolctl: 3.6.0
Built with OpenMP: True
threadpoolctl info:
user_api: openmp
internal_api: openmp
num_threads: 4
prefix: libgomp
filepath: /home/killer/.local/lib/python3.13/site-packages/scikit_learn.libs/libgomp-e985bcbb.so.1.0.0
version: None
Interest in fixing the bug
Proposed Solution:
I would like to investigate the check_array / dtype handling inside StandardScaler.transform to ensure it respects the input dtype without silently upcasting, provided the user explicitly inputs float32.
If the maintainers agree this is an anti-pattern, please assign this to me. I am happy to open a PR to patch the data-copying behavior.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with StandardScaler.transform and its check_array/dtype handling, using the provided float32 Pipeline reproduction to inspect the output dtype and memory use. Done means the expected float32 output is preserved without the reported upcast, with regression coverage for this behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100