dotnet / dotnet/machinelearning

DetectIidSpike not finding spikes ...

Open
#5,981 3 comments 0 reactions 0 assignees View on GitHub
area-TimeSeries
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

- OS & Version: Windows 10 Pro 64 bit
- ML.NET Version: 1.6.0
- .NET Version: .NET 5.0.402

I forked the machinelearning-samples repository. I generated a new product-sales.csv with 250 rows (instead of 36), and random values between 20K-30K.

There is one row in the file (at line 53) with an obvious spike of 60K.

Running the program does not pick up the spike.

**To Reproduce**
Using the machine-learning-samples code:
1. Replace the product-sales.csv with the one above: https://github.com/FlukeFan/machinelearning-samples/blob/main/samples/csharp/end-to-end-apps/AnomalyDetection-Sales/SpikeDetectionE2EApp/Data/product-sales.csv#L53

2. Set the number of entries in the file to 250: https://github.com/FlukeFan/machinelearning-samples/blob/main/samples/csharp/end-to-end-apps/AnomalyDetection-Sales/SpikeDetectionE2EApp/SpikeDetection.ModelTrainer/Program.cs#L29

3. Run the program - no spike is detected.

**Expected behavior**
I would expect the DetectIidSpike transform to identify the spike.

**Screenshots, Code, Sample Projects**
The code I used is a fork here: https://github.com/FlukeFan/machinelearning-samples/tree/main/samples/csharp/end-to-end-apps/AnomalyDetection-Sales

There was a single commit to generate the test data: https://github.com/FlukeFan/machinelearning-samples/commit/d838f17b16d5cce71397416a9481711e6089c600

**Additional context**
I notice that reducing the pvalueHistoryLength to about 50 highlights the spike, however I would expect larger values of the sliding window to also pick up the spike.

I had originally observed this problem when trying the spike detection on a much much larger file (many thousands of records), but I have reduced it to the 250 line file the demonstrates the problem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.