save_model fails to save the model after a certain number of rounds irrespective of the number of observations
- Dominant language
- C++
- Stars
- 9.1k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Problem:
The `save_model` method fails to save the model after a certain number of rounds irrespective of the number of observations;
although the file itself is created, its size is 0 KB if `n_rounds` is set to a higher integer than _10910_. The `train_pool` dimensions are:
```
dim(training) # note: 'training' is used to create the 'train_pool' for use in the code below
[1] 2647608 100
```
The successfully and empty saved models:

However I would like to point out that I tried it with a lower and bigger dataset (+- 50% of the number of rows), it does not seem to influence the aforementioned outcome (my dataset contains many similar observations so that would make sense to a certain degree). I suspect it has something to do with model complexity, so as the model gets progressively bigger in size, it somehow fails to be saved after a certain threshold is reached.
Code:
```
import gc
import time
import pandas as pd
import numpy as np
from catboost import CatBoostClassifier, Pool
import os
import re
def change_to_latest_processed_dir():
# Start from the root directory where the script is located
root_directory = os.getcwd()
# Define the path to the 'data' directory
data_directory = os.path.join(root_directory, "data")
# Check if the 'data' directory exists
if not os.path.exists(data_directory):
raise Exception("The 'data' directory does not exist in the root directory")
# List all directories in the 'data' directory
all_folders = [os.path.join(data_directory, d) for d in os.listdir(data_directory) if os.path.isdir(os.path.join(data_directory, d))]
# Filter directories starting with "processed_2"
processed_folders = [d for d in all_folders if re.match("^processed_2", os.path.basename(d))]
# If there are no processed folders, stop the script
if not processed_folders:
raise Exception("No 'processed_2*' folders found in the 'data' directory")
# Get the latest modified folder
latest_folder = max(processed_folders, key=os.path.getmtime)
# Change to the latest modified folder
os.chdir(latest_folder)
print(f"Changed directory to {latest_folder}")
# Call the function to change the directory
change_to_latest_processed_dir()
# Sample the middle bucket to lower the number of observations
trial = 1
var_to_predict = "ratio_return_group"
num_rounds = 10910 # 11500 fails
manual_cw = True
auto_cw = "None" # SqrtBalanced / Balanced / None
metric = "MultiClass" # MultiClass / MultiClassOneVsAll
# Load the pools
train_pool = Pool("cat_training.pool", column_description="cat_column_description.cd")
# Set seed for reproducibility
np.random.seed(777)
# Train CatBoost model
if manual_cw:
class_weights = [1, 1, 1, 2, 2] # Adjust as needed
model = CatBoostClassifier(iterations=num_rounds,
learning_rate=0.01,
depth=12,
l2_leaf_reg=1.017,
subsample = 0.81, # Sample rate for bagging (row subsampling)
bootstrap_type = 'Bernoulli',
loss_function=metric,
eval_metric=metric,
task_type='GPU', # or 'CPU' if GPU support is not available
class_weights=class_weights)
else:
model = CatBoostClassifier(iterations=num_rounds,
learning_rate=0.01,
depth=12,
l2_leaf_reg=1.017,
subsample = 0.81, # Sample rate for bagging (row subsampling)
bootstrap_type = 'Bernoulli',
loss_function=metric,
eval_metric=metric,
task_type='GPU', # or 'CPU' if GPU support is not available
auto_class_weights=auto_cw)
model.fit(train_pool)
time.sleep(3)
gc.collect()
class_weights_str = ''.join(map(str, class_weights)) if manual_cw else '0' # Convert class_weights to a string of concatenated numbers
# Construct the filename
model_filename = f"depth_trades_catboo_fact_trial_{trial}_nrounds_{num_rounds}_{var_to_predict}_manual_cw_{manual_cw}_{class_weights_str}_auto_cw_{auto_cw}_{metric}.cbm"
# Save the model
model.save_model(model_filename)
```
Setting either `auto_cw=True` or `class_weights` does not influence the outcome.
The result is consistent across 2 different machines with different GPUs and amount of RAM; also, both machines result in the same outcome regardless of the program running on Windows or WSL Debian.
PS The comparable R code results in the same outcome too.
PPS I don't want to try and train on CPU, it will take 10+ days.
Catboost version: 1.2.2
Operating System:
- Windows 11 Pro
- Debian GNU/Linux 12 (bookworm)
CPU: Intel(R) Core(TM) i9-14900K 3.20 GHz
GPU:
- NVIDIA GeForce RTX 3080 `# thought the problem could be my GPU / 16GB RAM, hence the upgrade to 64GB RAM & RTX 4090...`
- NVIDIA GeForce RTX 4090 `# 64GB RAM`
Contributor guide
Research direction
Start with the provided Python reproduction, especially CatBoostClassifier.fit and model.save_model, using the supplied pool and column description. Compare model files at n_rounds 10910 and 11500 on GPU, then verify that saving produces a non-empty model in both Python and the comparable R case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, r
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100