catboost / catboost/catboost

save_model fails to save the model after a certain number of rounds irrespective of the number of observations

Open
#2,564 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
9.1k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Problem:
The `save_model` method fails to save the model after a certain number of rounds irrespective of the number of observations;
although the file itself is created, its size is 0 KB if `n_rounds` is set to a higher integer than _10910_. The `train_pool` dimensions are:
```
dim(training) # note: 'training' is used to create the 'train_pool' for use in the code below
[1] 2647608 100
```

The successfully and empty saved models:

![catboo_n_rounds](https://github.com/catboost/catboost/assets/54410381/ebbcc536-6710-4118-a813-f441d966ba79)

However I would like to point out that I tried it with a lower and bigger dataset (+- 50% of the number of rows), it does not seem to influence the aforementioned outcome (my dataset contains many similar observations so that would make sense to a certain degree). I suspect it has something to do with model complexity, so as the model gets progressively bigger in size, it somehow fails to be saved after a certain threshold is reached.

Code:
```
import gc
import time
import pandas as pd
import numpy as np
from catboost import CatBoostClassifier, Pool
import os
import re

def change_to_latest_processed_dir():
# Start from the root directory where the script is located
root_directory = os.getcwd()

# Define the path to the 'data' directory
data_directory = os.path.join(root_directory, "data")

# Check if the 'data' directory exists
if not os.path.exists(data_directory):
raise Exception("The 'data' directory does not exist in the root directory")

# List all directories in the 'data' directory
all_folders = [os.path.join(data_directory, d) for d in os.listdir(data_directory) if os.path.isdir(os.path.join(data_directory, d))]

# Filter directories starting with "processed_2"
processed_folders = [d for d in all_folders if re.match("^processed_2", os.path.basename(d))]

# If there are no processed folders, stop the script
if not processed_folders:
raise Exception("No 'processed_2*' folders found in the 'data' directory")

# Get the latest modified folder
latest_folder = max(processed_folders, key=os.path.getmtime)

# Change to the latest modified folder
os.chdir(latest_folder)
print(f"Changed directory to {latest_folder}")

# Call the function to change the directory
change_to_latest_processed_dir()

# Sample the middle bucket to lower the number of observations
trial = 1
var_to_predict = "ratio_return_group"
num_rounds = 10910 # 11500 fails
manual_cw = True
auto_cw = "None" # SqrtBalanced / Balanced / None
metric = "MultiClass" # MultiClass / MultiClassOneVsAll

# Load the pools
train_pool = Pool("cat_training.pool", column_description="cat_column_description.cd")

# Set seed for reproducibility
np.random.seed(777)

# Train CatBoost model
if manual_cw:
class_weights = [1, 1, 1, 2, 2] # Adjust as needed
model = CatBoostClassifier(iterations=num_rounds,
learning_rate=0.01,
depth=12,
l2_leaf_reg=1.017,
subsample = 0.81, # Sample rate for bagging (row subsampling)
bootstrap_type = 'Bernoulli',
loss_function=metric,
eval_metric=metric,
task_type='GPU', # or 'CPU' if GPU support is not available
class_weights=class_weights)

else:
model = CatBoostClassifier(iterations=num_rounds,
learning_rate=0.01,
depth=12,
l2_leaf_reg=1.017,
subsample = 0.81, # Sample rate for bagging (row subsampling)
bootstrap_type = 'Bernoulli',
loss_function=metric,
eval_metric=metric,
task_type='GPU', # or 'CPU' if GPU support is not available
auto_class_weights=auto_cw)

model.fit(train_pool)
time.sleep(3)
gc.collect()

class_weights_str = ''.join(map(str, class_weights)) if manual_cw else '0' # Convert class_weights to a string of concatenated numbers
# Construct the filename
model_filename = f"depth_trades_catboo_fact_trial_{trial}_nrounds_{num_rounds}_{var_to_predict}_manual_cw_{manual_cw}_{class_weights_str}_auto_cw_{auto_cw}_{metric}.cbm"

# Save the model
model.save_model(model_filename)
```
Setting either `auto_cw=True` or `class_weights` does not influence the outcome.

The result is consistent across 2 different machines with different GPUs and amount of RAM; also, both machines result in the same outcome regardless of the program running on Windows or WSL Debian.

PS The comparable R code results in the same outcome too.
PPS I don't want to try and train on CPU, it will take 10+ days.

Catboost version: 1.2.2

Operating System:
- Windows 11 Pro
- Debian GNU/Linux 12 (bookworm)

CPU: Intel(R) Core(TM) i9-14900K 3.20 GHz

GPU:
- NVIDIA GeForce RTX 3080 `# thought the problem could be my GPU / 16GB RAM, hence the upgrade to 64GB RAM & RTX 4090...`
- NVIDIA GeForce RTX 4090 `# 64GB RAM`

Contributor guide

Open the contributing guide

Research direction

Start with the provided Python reproduction, especially CatBoostClassifier.fit and model.save_model, using the supplied pool and column description. Compare model files at n_rounds 10910 and 11500 on GPU, then verify that saving produces a non-empty model in both Python and the comparable R case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.