dotnet / dotnet/machinelearning-samples

Problem with DetectIidSpike

Open
#816 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
4.7k
Forks
2.7k
Avg merge
2d 22h
Merged PRs (30d)
1

Description

I want to create a spike detector as you have demonstrated in its sample page. Here it is my code

```
class Program
{
private static MLContext mlContext;
static void Main(string[] args)
{
mlContext = new MLContext();

//assign the Number of records in dataset file to cosntant variable
const int size = 36;

//Load the data into IDataView.
//This dataset is used while prediction/detecting spikes or changes.
IDataView dataView = mlContext.Data.LoadFromTextFile(path: "product-sales.csv", hasHeader: true, separatorChar: ',');

//To detech temporay changes in the pattern
DetectSpike(size, dataView);

Console.WriteLine("=============== End of process, hit any key to finish ===============");

Console.ReadLine();
}

static void DetectSpike(int size, IDataView dataView)
{
Console.WriteLine("===============Detect temporary changes in pattern===============");

//STEP 1: Create Esimtator
var estimator = mlContext.Transforms.DetectIidSpike(outputColumnName: nameof(ProductSalesPrediction.Prediction), inputColumnName: nameof(ProductSalesData.numSales), confidence: 95, pvalueHistoryLength: size / 4);

//STEP 2:The Transformed Model.
//In IID Spike detection, we don't need to do training, we just need to do transformation.
//As you are not training the model, there is no need to load IDataView with real data, you just need schema of data.
//So create empty data view and pass to Fit() method.
ITransformer tansformedModel = estimator.Fit(CreateEmptyDataView());

//STEP 3: Use/test model
//Apply data transformation to create predictions.
IDataView transformedData = tansformedModel.Transform(dataView);
var predictions = mlContext.Data.CreateEnumerable(transformedData, reuseRowObject: false);

Console.WriteLine("Alert\tScore\tP-Value");
foreach (var p in predictions)
{
if (p.Prediction[0] == 1)
{
Console.BackgroundColor = ConsoleColor.DarkYellow;
Console.ForegroundColor = ConsoleColor.Black;
}
Console.WriteLine("{0}\t{1:0.00}\t{2:0.00}", p.Prediction[0], p.Prediction[1], p.Prediction[2]);
Console.ResetColor();
}
Console.WriteLine("");
}

private static IDataView CreateEmptyDataView()
{
//Create empty DataView. We just need the schema to call fit()
IEnumerable enumerableData = new List();
var dv = mlContext.Data.LoadFromEnumerable(enumerableData);
return dv;
}
}

public class ProductSalesData
{
[LoadColumn(0)]
public string Month;

[LoadColumn(1)]
public float numSales;
}

public class ProductSalesPrediction
{
//vector to hold alert,score,p-value values
[VectorType(3)]
public double[] Prediction { get; set; }
}
```
I have attached the dataset and at line 6, the numSales value is "703.5" which is absolutely a spike but is not detected as a spike. This is my result:
```
===============Detect temporary changes in pattern===============
Alert Score P-Value
0 271.00 0.50
0 150.90 0.00
0 188.10 0.41
0 124.30 0.13
0 185.30 0.47
0 703.50 0.00 --> This line is not detected as a spike
0 236.80 0.50
0 229.50 0.49
0 197.80 0.44
0 127.90 0.30
0 341.50 0.27
0 190.90 0.42
0 199.30 0.44
0 154.50 0.33
0 215.10 0.46
0 278.30 0.19
0 196.40 0.43
0 292.00 0.17
0 231.00 0.45
0 308.60 0.18
0 294.90 0.19
**1 426.60 0.00**
0 269.50 0.47
0 347.30 0.21
0 344.70 0.27
0 445.40 0.06
0 320.90 0.49
0 444.30 0.12
0 406.30 0.29
0 442.40 0.21
**1 580.50 0.00**
0 412.60 0.45
**1 687.00 0.01**
0 480.30 0.40
0 586.30 0.20
0 651.90 0.14

=============== End of process, hit any key to finish ===============
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.