dotnet / dotnet/machinelearning
DetectIidSpike not finding spikes ...
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
- OS & Version: Windows 10 Pro 64 bit
- ML.NET Version: 1.6.0
- .NET Version: .NET 5.0.402
I forked the machinelearning-samples repository. I generated a new product-sales.csv with 250 rows (instead of 36), and random values between 20K-30K.
There is one row in the file (at line 53) with an obvious spike of 60K.
Running the program does not pick up the spike.
**To Reproduce**
Using the machine-learning-samples code:
1. Replace the product-sales.csv with the one above: https://github.com/FlukeFan/machinelearning-samples/blob/main/samples/csharp/end-to-end-apps/AnomalyDetection-Sales/SpikeDetectionE2EApp/Data/product-sales.csv#L53
2. Set the number of entries in the file to 250: https://github.com/FlukeFan/machinelearning-samples/blob/main/samples/csharp/end-to-end-apps/AnomalyDetection-Sales/SpikeDetectionE2EApp/SpikeDetection.ModelTrainer/Program.cs#L29
3. Run the program - no spike is detected.
**Expected behavior**
I would expect the DetectIidSpike transform to identify the spike.
**Screenshots, Code, Sample Projects**
The code I used is a fork here: https://github.com/FlukeFan/machinelearning-samples/tree/main/samples/csharp/end-to-end-apps/AnomalyDetection-Sales
There was a single commit to generate the test data: https://github.com/FlukeFan/machinelearning-samples/commit/d838f17b16d5cce71397416a9481711e6089c600
**Additional context**
I notice that reducing the pvalueHistoryLength to about 50 highlights the spike, however I would expect larger values of the sliding window to also pick up the spike.
I had originally observed this problem when trying the spike detection on a much much larger file (many thousands of records), but I have reduced it to the 250 line file the demonstrates the problem.
Contributor guide
Assessment
This issue has not been assessed yet.