Instance Weighting

🚀 Feature

Support providing weights for instances (not classes).

Motivation

As is the case with search engines, I have a dataset in which a document (e.g., query) might appear more than once. From a binary relevance stand point (i.e., the classification task), all occurrences of the same document have the same label. To save some memory and computation, instead of having a unique document d repeated N_d times in this dataset, and using those repeated instances to compute the relevant metrics (e.g., F_1 score), it's better to provide the array of weights as a parameter to use it as a multiplier for TP, TN, FP and FN.

Pitch

Add a parameter example_weight.

Alternatives

Repeat the documents N times. But this is not practical when some documents appear several orders of magnitudes compared to others (e.g., head vs tail queries in a search engine).

No files or tests are named. Start by locating the classification metric implementations and their handling of TP, TN, FP, and FN, then determine which metric APIs should accept example_weight. Done means instance weights are supported consistently and covered by tests for weighted confusion-matrix-derived metrics.