catboost / catboost/catboost

Bad allocation when using CatBoost with AutoGluon

Open
#3,072 5 comments 0 reactions 0 assignees View on GitHub
python Windows
Dominant language
C++
Stars
9.1k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug
When using the CatBoost Python package on a system with 16GB of RAM (common in developer laptops, e.g., with GTX 1650 Ti), `CatBoostError: bad allocation` occurs frequently, especially during AutoML pipelines (like AutoGluon) or multi-process training.
Investigation of the source code reveals that `ConfigureMalloc` in `catboost/private/libs/algo/helpers.cpp` explicitly sets the `lfalloc` cache limit to **1,000,000 pages (~4GB)**, regardless of the system's total physical RAM or the presence of other concurrent processes.
### Code Analysis
The root cause seems to be in `catboost/private/libs/algo/helpers.cpp`:
```cpp
void ConfigureMalloc() {
#if !(defined(__APPLE__) && defined(__MACH__)) && !defined(__aarch64__)
NMalloc::MallocInfo().SetParam("LB_LIMIT_TOTAL_SIZE", "1000000"); // Hardcoded ~4GB cache
#endif
}
```
This configuration is significantly higher than the default 500 * 1024 * 1024 / 4096 (~500MB) defined in library/cpp/lfalloc/lf_allocX64.h.

In environments like AutoGluon, where multiple models may be initialized or multiple processes may be running, each CatBoost process "sequesters" up to 4GB of RAM in its internal allocator cache. This leads to premature std::bad_alloc when the OS cannot fulfill new allocation requests, even if physical RAM hasn't been fully utilized by actual data yet.

Furthermore, calling get_gpu_device_count() triggers TDevicesProvider::Initilize(), which spawns worker threads for every GPU found (e.g., GTX 1650 Ti), adding additional memory overhead and CUDA context initialization even when task_type='CPU' is intended.

Environment

OS: Windows 11(x64)
RAM: 32GB
GPU: NVIDIA GTX 1650 Ti (4GB VRAM)
Library versions: CatBoost 1.2.10, AutoGluon 1.5

To Reproduce

Run two concurrent Python processes using AutoGluon or CatBoost with moderate datasets.

Observe WorkingSet vs VirtualMemorySize.

Exception bad_alloc is eventually thrown during Pool creation or at the start of fit().

Suggested Improvement

Configurability: Allow LB_LIMIT_TOTAL_SIZE to be set via an environment variable (e.g., CATBOOST_ALLOC_CACHE_LIMIT) or a global training parameter.

RAM-Aware Defaults: Scale the cache limit based on available physical RAM at runtime instead of using a hardcoded constant.

Lazy GPU Init: Delay thread/worker creation in TDevicesProvider until a GPU task is actually requested, or provide a way to check for GPU presence without full initialization.

test_memory.py:
```python
import catboost
import os
import psutil
import time
import sys
from catboost import _catboost
import numpy as np

def get_process_memory():
process = psutil.Process(os.getpid())
return process.memory_info().rss / (1024 * 1024)

print(f"Initial Memory: {get_process_memory():.2f} MB")

print("\n--- Testing get_gpu_device_count() ---")
sys.stdout.flush()
try:
gpu_count = _catboost._get_gpu_device_count()
print(f"GPU Count: {gpu_count}")
except Exception as e:
print(f"GPU Count failed: {e}")
print(f"Memory after get_gpu_device_count: {get_process_memory():.2f} MB")

print("\n--- Testing Model Initialization ---")
sys.stdout.flush()
try:
model = catboost.CatBoostClassifier(iterations=1, task_type='CPU', silent=True)
model.fit([[1, 2], [3, 4], [5, 6]], [0, 1, 0])
print("Tiny model fit successful")
except Exception as e:
print(f"Tiny model fit failed: {e}")
print(f"Memory after tiny fit: {get_process_memory():.2f} MB")

print("\n--- Testing Pool allocation (1GB) ---")
sys.stdout.flush()
try:
# 25M float32 values = 100MB
# 25M * 10 cols = 250M float32 = 1GB
rows = 25_000_000
cols = 10
print(f"Creating 1GB numpy array...")
data = np.random.random((rows, cols)).astype(np.float32)
labels = np.random.randint(0, 2, rows).astype(np.float32)
print(f"Numpy Memory (RSS): {get_process_memory():.2f} MB")

print("Creating CatBoost Pool...")
sys.stdout.flush()
pool = catboost.Pool(data, labels)
print(f"Pool created. Memory (RSS): {get_process_memory():.2f} MB")

print("Deleting numpy data to see if Pool keeps its own copy...")
del data
import gc
gc.collect()
print(f"Memory after del data: {get_process_memory():.2f} MB")

print("Running a tiny fit on 1GB pool...")
sys.stdout.flush()
model = catboost.CatBoostClassifier(iterations=1, task_type='CPU', silent=True)
model.fit(pool)
print(f"Fit completed. Memory (RSS): {get_process_memory():.2f} MB")

except Exception as e:
print(f"Caught Exception in Pool test: {type(e).__name__}: {e}")

print("\n--- Final Summary ---")
print(f"Final RSS: {get_process_memory():.2f} MB")

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing test_memory.py and tracing ConfigureMalloc in catboost/private/libs/algo/helpers.cpp alongside the default in library/cpp/lfalloc/lf_allocX64.h. Separately inspect the get_gpu_device_count() path and TDevicesProvider initialization to determine whether allocator limits and GPU setup are related. Done requires narrowing this broad report to one confirmed, maintainer-approved behavior change with a reproducible test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, machine-learning, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.