Additional Sampling Method: The Bootstrap

During the previous OpenML Workshop, the Bootstrap was proposed as an additional sampling method in OpenML. Today I started programming it, and some issues came up. Instead of discussing it privately, I think it's a good idea to put these up here.

As reference, we used the Data Mining book by Witten and Frank, (pdf available here: http://177.101.20.73/docs/wittenfrank.pdf) page 152 (186 when including roman page numbers) is about Bootstrapping.

1. First of all, they estimate the accuracy by means of a formula which involves a high proportion (63.2%) of the accuracy obtained on test instances that do not occur in the training set, and a small proportion (36.8%) of the accuracy obtained on instances that do occur in the trainings set. The last part seems counter intuitive to me. Most of the things I have learned in Machine Learning is that estimating performance over trained instances is very bad. I assume we still want to stick to this definition, since it appears to be accepted by the community.

2. Based upon the decision made for point one, there is an additional technical question. So assume that we obtain the error rate by the following definition:

e = 0.386 * e_l + 0.632 * e_1 (where e_0 is the error rate on unseen examples, and e_1 is the error rate on all other examples).

Does this definition generalize to other evaluation measures we attempt to store in OpenML? For example, kappa statistic, recall, or even auroc? It doesn't seem to make complete sense to me.