dotnet / dotnet/machinelearning
[AutoML] Reservoir sample dataset statistics
Open
area-AutoML
enhancement
Priority:2
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
Currently dataset statistics within AutoML are calculated from the first 1,000 rows of a dataset. Instead, we should be calculating statistics from a random sample of 1,000 rows. (First 1,000 rows could be biased if they are sorted by label, any other column, time of collection, etc.) We can use reservoir sampling to obtain a random sample of a fixed size in a single pass over the dataset
Contributor guide
Assessment
This issue has not been assessed yet.