Chpt.7 Ensemble Methods - AdaBoost

Hi,

I am currently working through the ensemble methods (chapter 7) in the book and I wanted to code out the AdaBoost algorithm from first principles. In the book, there is code that shows 5 SVM classifiers being tained consecutively on the moons dataset using AdaBoost technique. However, the weight update in the code is:

sample_weights[y_pred != y_train] *= (1 + learning_rate)

where as in the book the weight update is quoted as:

sample_weights[y_pred != y_train] *= np.exp(alpha_j)

Firstly, why in the book are the updates being calculated as they are and not in using the formulas shown in the book?

My second issue is that I wanted to re code the SVM classifiers using the equations in the book to see how the adaBoost works. I have written the following code:

## Implimenting Adaboost (1st Principles)

m = len(X_train)
sample_weights = np.ones(m) #initialise at 1 (in book it does say that each weight should be initialised at 1/m)
learning_rate = 1
models = {}
r_js = []
alphas = []

for i in range(5):
print("iteration {0}".format(i))

svm_clf = SVC(kernel="rbf", C=0.05, gamma="scale", random_state=42)
svm_clf = svm_clf.fit(X_train, y_train, sample_weight=sample_weights)

models["SVM_"+str(i)] = svm_clf #storing the SVM trained models
y_pred = svm_clf.predict(X_train)

r_j = sample_weights[y_pred != y_train].sum() / sample_weights.sum()
r_js.append(r_j)

number = (1-r_j)/r_j
alpha_j = learning_rate * np.log10(number)
alphas.append(alpha_j)
sample_weights[y_pred != y_train] *= np.exp(alpha_j)

print(len(sample_weights[y_pred != y_train]))

#sample_weights /= sample_weights.sum() #normalising the sample weights by dividing by the sum of the weights

Running the code, the weights do get altered as expected and I get the following misclassifications for 5 models:

iteration 0
49
iteration 1
41
iteration 2
32
iteration 3
36
iteration 4
46

this looks like it is working (until the 5th models) as the classifications are getting better.

I then did the initialisation properly by setting the weights to be 1/m using:

sample_weights = np.ones(m)/m

and I normalised by the sum of the weights at the end of each iteration with,

sample_weights /= sample_weights.sum()

When I impliment these two lines of code I get the following misclassification rate:

iteration 0
193
iteration 1
193
iteration 2
193
iteration 3
193
iteration 4
193

showing that there is just not change in the classification.

- Is this the correct code for the implimentation?
- does the svm_clf.fit() method with the param sample_weights do what I am intending it to do?

Any help would be greatly appreciated here!

thanks,
Dan