dmlc / dmlc/xgboost

Bootstrap Confidence Intervals for XGBoost regression (Python)

Open
#5,475 5 comments 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

I want to construct Bootstrap Confidence Intervals for XGBoost regression using python. I developed my case based on codes (https://machinelearningmastery.com/calculate-bootstrap-confidence-intervals-machine-learning-results-python/#comment-528118).
Question: I am getting a one bin histogram. I get the single value for the score when we do n_iterations for the bootstrap. This is the problem and it is related to the way I am getting RMSE. Though I tried to find RMSE in different ways. yet, I could not solve the problem How can we solve it?

![image](https://user-images.githubusercontent.com/56795978/78259655-fb816b00-74ca-11ea-92b1-99e194e1a6f2.png)

import numpy
from pandas import read_csv
from sklearn.datasets import load_boston
from sklearn.utils import resample
from matplotlib import pyplot
from xgboost import XGBRegressor
import pandas as pd
import numpy as np
from sklearn.metrics import mean_squared_error
# load dataset
boston_dataset = load_boston()

df = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)

df['MEDV'] = boston_dataset.target
values1 = df.values
# configure bootstrap
n_iterations = 1000
n_size = int(len(df) * 0.50)
# run bootstrap
stats = list()

# prepare train and test sets

for i in range(n_iterations):
# prepare train and test sets
train = resample(values1, n_samples=n_size)
test = numpy.array([x for x in values1 if x.tolist() not in train.tolist()])

model = XGBRegressor() ## Final for the papers


X_train = train[:,:-1]
y_train = train[:,-1]
X_test = test[:,:-1]
y_test = test[:,-1]

model.fit(X_train,y_train)
predictions = model.predict(X_test)
# make predictions

def rmse_calculator(predicted, actual):

assert len(predicted) == len(actual)
return np.sqrt(
np.mean(
np.power(predicted- actual, 2)))
score = rmse_calculator(y_test , predictions)

#score = mean_squared_error(y_test, predictions) ** 0.5
yt = np.asarray(y_test)
y_pred = np.asarray(predictions)
score = np.sqrt(mean_squared_error(yt,y_pred))
print(score)
stats.append(score)
# plot scores
pyplot.hist(stats)
pyplot.show()
# confidence intervals
alpha = 0.95
p = ((1.0-alpha)/2.0) * 100
lower = max(0.0, numpy.percentile(stats, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, numpy.percentile(stats, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100))

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.