dotnet / dotnet/machinelearning
Enable Binary Classification Metric Calculation on Huge Datasets
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
From experimenting, it seems like calculating binary classification metrics does not scale to huge datasets. Taking a heap dump to examine the high memory usage (before the program runs out of memory), I see a list of floats used by `UnweightedAucAggregator`. It looks like, to calculate AUC, every prediction is kept in memory. It also looks like there is already substantial logic to account for this scenario -- there's logic to reservoir sample predictions, and then calculate AUC on the sample. However, it looks like the size of the internal parameter `MaxAucExamples` to control the size of this reservoir sample is always set to -1, and not exposed to the end user?https://github.com/dotnet/machinelearning/blob/610ffcb67083c2e5e6e1a14884ba24b1da0384c7/src/Microsoft.ML.Data/Evaluators/BinaryClassifierEvaluator.cs#L45
Perhaps we should somehow expose this parameter to enable binary metric calculation on huge datasets, or set the parameter to some reasonable default
@justinormont, @vinodshanbhag
Contributor guide
Assessment
This issue has not been assessed yet.