Bootstrap Inference and Sample Weights

When using sample weights, the current bootstrap is no longer reasonable. Consider the case of one binary feature and were data are summarized and consist of means conditional on each feature value, with sample_weight corresponding to the number of original observations used to calculate the mean. Then our summarized data set consists of two samples each with weights. The bootstrap will be non-sense here as it will be drawing sometimes one sample twice or both samples. That’s not the intention. We somehow need to simulate what bootstrap would have done in the original sample.

This is not possible simply with sample weights. Ideally we would need access to all original samples. One idea: also solicit sample variances (i.e. variances within segment); which will be stored in the sample_vars fit argument. Then what bootstrap can do is treat each segment as distributed from a normal with the given mean and variance. Then draw fresh random samples from that normal. Moreover the number of samples that we draw from each segment should be determined as follows: we draw n segment ids each with probability proportional to the original number of samples in that segment. Then for each of these segment samples we draw a y value drawn from the normal approximation of that segment.

For now, the package correctly throws a warning that “ the bootstrap is used with sample weights and we advise against as the current implementation of bootstrap resampling does not take into account sample weights.”