Bad allocation when using CatBoost with AutoGluon
- Dominant language
- C++
- Stars
- 9.1k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
When using the CatBoost Python package on a system with 16GB of RAM (common in developer laptops, e.g., with GTX 1650 Ti), `CatBoostError: bad allocation` occurs frequently, especially during AutoML pipelines (like AutoGluon) or multi-process training.
Investigation of the source code reveals that `ConfigureMalloc` in `catboost/private/libs/algo/helpers.cpp` explicitly sets the `lfalloc` cache limit to **1,000,000 pages (~4GB)**, regardless of the system's total physical RAM or the presence of other concurrent processes.
### Code Analysis
The root cause seems to be in `catboost/private/libs/algo/helpers.cpp`:
```cpp
void ConfigureMalloc() {
#if !(defined(__APPLE__) && defined(__MACH__)) && !defined(__aarch64__)
NMalloc::MallocInfo().SetParam("LB_LIMIT_TOTAL_SIZE", "1000000"); // Hardcoded ~4GB cache
#endif
}
```
This configuration is significantly higher than the default 500 * 1024 * 1024 / 4096 (~500MB) defined in library/cpp/lfalloc/lf_allocX64.h.
In environments like AutoGluon, where multiple models may be initialized or multiple processes may be running, each CatBoost process "sequesters" up to 4GB of RAM in its internal allocator cache. This leads to premature std::bad_alloc when the OS cannot fulfill new allocation requests, even if physical RAM hasn't been fully utilized by actual data yet.
Furthermore, calling get_gpu_device_count() triggers TDevicesProvider::Initilize(), which spawns worker threads for every GPU found (e.g., GTX 1650 Ti), adding additional memory overhead and CUDA context initialization even when task_type='CPU' is intended.
Environment
OS: Windows 11(x64)
RAM: 32GB
GPU: NVIDIA GTX 1650 Ti (4GB VRAM)
Library versions: CatBoost 1.2.10, AutoGluon 1.5
To Reproduce
Run two concurrent Python processes using AutoGluon or CatBoost with moderate datasets.
Observe WorkingSet vs VirtualMemorySize.
Exception bad_alloc is eventually thrown during Pool creation or at the start of fit().
Suggested Improvement
Configurability: Allow LB_LIMIT_TOTAL_SIZE to be set via an environment variable (e.g., CATBOOST_ALLOC_CACHE_LIMIT) or a global training parameter.
RAM-Aware Defaults: Scale the cache limit based on available physical RAM at runtime instead of using a hardcoded constant.
Lazy GPU Init: Delay thread/worker creation in TDevicesProvider until a GPU task is actually requested, or provide a way to check for GPU presence without full initialization.
test_memory.py:
```python
import catboost
import os
import psutil
import time
import sys
from catboost import _catboost
import numpy as np
def get_process_memory():
process = psutil.Process(os.getpid())
return process.memory_info().rss / (1024 * 1024)
print(f"Initial Memory: {get_process_memory():.2f} MB")
print("\n--- Testing get_gpu_device_count() ---")
sys.stdout.flush()
try:
gpu_count = _catboost._get_gpu_device_count()
print(f"GPU Count: {gpu_count}")
except Exception as e:
print(f"GPU Count failed: {e}")
print(f"Memory after get_gpu_device_count: {get_process_memory():.2f} MB")
print("\n--- Testing Model Initialization ---")
sys.stdout.flush()
try:
model = catboost.CatBoostClassifier(iterations=1, task_type='CPU', silent=True)
model.fit([[1, 2], [3, 4], [5, 6]], [0, 1, 0])
print("Tiny model fit successful")
except Exception as e:
print(f"Tiny model fit failed: {e}")
print(f"Memory after tiny fit: {get_process_memory():.2f} MB")
print("\n--- Testing Pool allocation (1GB) ---")
sys.stdout.flush()
try:
# 25M float32 values = 100MB
# 25M * 10 cols = 250M float32 = 1GB
rows = 25_000_000
cols = 10
print(f"Creating 1GB numpy array...")
data = np.random.random((rows, cols)).astype(np.float32)
labels = np.random.randint(0, 2, rows).astype(np.float32)
print(f"Numpy Memory (RSS): {get_process_memory():.2f} MB")
print("Creating CatBoost Pool...")
sys.stdout.flush()
pool = catboost.Pool(data, labels)
print(f"Pool created. Memory (RSS): {get_process_memory():.2f} MB")
print("Deleting numpy data to see if Pool keeps its own copy...")
del data
import gc
gc.collect()
print(f"Memory after del data: {get_process_memory():.2f} MB")
print("Running a tiny fit on 1GB pool...")
sys.stdout.flush()
model = catboost.CatBoostClassifier(iterations=1, task_type='CPU', silent=True)
model.fit(pool)
print(f"Fit completed. Memory (RSS): {get_process_memory():.2f} MB")
except Exception as e:
print(f"Caught Exception in Pool test: {type(e).__name__}: {e}")
print("\n--- Final Summary ---")
print(f"Final RSS: {get_process_memory():.2f} MB")
```
Contributor guide
Research direction
Start by reproducing test_memory.py and tracing ConfigureMalloc in catboost/private/libs/algo/helpers.cpp alongside the default in library/cpp/lfalloc/lf_allocX64.h. Separately inspect the get_gpu_device_count() path and TDevicesProvider initialization to determine whether allocator limits and GPU setup are related. Done requires narrowing this broad report to one confirmed, maintainer-approved behavior change with a reproducible test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, machine-learning, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100