dotnet / dotnet/machinelearning
System.OperationCanceledException after AutoML experiment
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
### System information
> .NET SDK (reflecting any global.json):
> Version: 5.0.103
> Commit: 72dec52dbd
>
> Runtime Environment:
> OS Name: Windows
> OS Version: 10.0.19042
> OS Platform: Windows
> RID: win10-x64
> Base Path: C:\Program Files\dotnet\sdk\5.0.103\
>
> Host (useful for support):
> Version: 5.0.3
> Commit: c636bbdc8a
>
> .NET SDKs installed:
> 5.0.103 [C:\Program Files\dotnet\sdk]
>
> .NET runtimes installed:
> Microsoft.AspNetCore.All 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.NETCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.WindowsDesktop.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
> Microsoft.WindowsDesktop.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
### Issue
After an experiment ended (TimeOut elapsed, with `token: CancellationToken.None`)
Sometimes the access of `experimentResult.BestRun.Estimator.Fit(_exDataView)` will raise `System.OperationCanceledException`.
- **What happened?**
```
System.OperationCanceledException: Operation was canceled.
at Microsoft.ML.Runtime.Contracts.CheckAlive(IHostEnvironment env)
at Microsoft.ML.Transforms.ValueToKeyMappingTransformer.Train(IHostEnvironment env, IChannel ch, ColInfo[] infos, IDataView keyData, ColumnOptionsBase[] columns, IDataView trainingData, Boolean autoConvert)
at Microsoft.ML.Transforms.ValueToKeyMappingTransformer..ctor(IHostEnvironment env, IDataView input, ColumnOptionsBase[] columns, IDataView keyData, Boolean autoConvert)
at Microsoft.ML.Transforms.ValueToKeyMappingEstimator.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
```
- **What did you expect?**
This exception does not occur all the time when accessing `BestRun` prop.
### Source code / logs
```
var experimentSettings = new MulticlassExperimentSettings
{
MaxExperimentTimeInSeconds = _maxExperimentTimeInSeconds,
CancellationToken = pManager.Token, // CancellationToken.None
OptimizingMetric = MulticlassClassificationMetric.MacroAccuracy
}
var experiment= _mlContext.Auto().CreateMulticlassClassificationExperiment(experimentSettings);
var experimentResult= experiment.Execute(_trainDataView, progressHandler: experimentProgress,
preFeaturizer: _mlContext.Transforms.DropColumns("LabelFeaturized"));
experimentResult.BestRun.Estimator.Fit(_trainDataView);
```
Contributor guide
Assessment
This issue has not been assessed yet.