dotnet / dotnet/machinelearning
IMonitor.ReportRunningTrial reports incorrect pipeline
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
**System Information (please complete the following information):**
- OS & Version: Windows 11 [Version 10.0.22621.675]
- ML.NET Version: ML.NT v2.0.0-preview.22525.3
- .NET Version: .NET 6.0.9
**Describe the bug**
I created a custom class that implements the `Microsoft.ML.AutoML.IMonitor` to monitor my training. It's `ReportRunningTrial` method has the following simple implementation:
`_logger.LogInformation($"{"Running".PadLeft(9)} Trial #{setting.TrialId.ToString().PadLeft(4)} - Pipeline: {_pipeline.ToString(setting.Parameter).PadRight(70)}");`
The other, `ReportBestTrial`, `ReportCompletedTrial` and `ReportFailTrial` methods look very similar.
When I run my training it looks like that the `ReportRunningTrial` method's pipeline object wasn't updated, so it doesn't show the pipeline of the current trial.
**To Reproduce**
Steps to reproduce the behavior:
1. Create [a custom class](https://github.com/andrasfuchs/BioBalanceDetector/blob/master/Software/Source/BBDProto08/BBD.BodyMonitor/BBD.BodyMonitor.API/Models/MLContextLoggerMonitor.cs) that implements IMonitor
2. Create an ML.NET experiment, and use that monitor class, like this:
```
var experiment = mlContext.Auto().CreateExperiment();
experiment
.SetPipeline(pipeline)
.SetTrainingTimeInSeconds(trainingTimeInSeconds)
.SetRegressionMetric(RegressionMetric.RSquared, labelColumn: "Label")
.SetDataset(trainTestData.TrainSet, trainTestData.TestSet)
.SetMonitor(monitor)
.SetMaximumMemoryUsageInMegaByte(20 * 1024);
```
3. Make sure that you allow more than one type of (regression) algorithms
4. Run the experiment
5. Check the output: you will see that the `settings.TrialId` property values look good, but the `_pipeline.ToString(setting.Parameter)` returns an invalid value in the `ReportRunningTrial` method.
**Expected behavior**
The `_pipeline.ToString(setting.Parameter)` should return the just-started trial's pipeline.
**Screenshots, Code, Sample Projects**

**Additional context**
You can see the problem on the screenshot:
Running Trial 0's pipeline looks like it was `FastTreeRegression`, but in reality it's `FastForestRegression`.
Running Trial 1's pipeline looks like it was `FastForestRegression`, but it's `FastTreeRegression`.
Running Trial 2's pipeline looks like it was `FastForestRegression`, but it's `LightGbmRegression`.
etc.
Edit: After testing a little more I'm not sure anymore which methods of the `IMonitor` interface report the pipeline wrong, it is possible that `ReportRunningTrial` is the correct one and the others are incorrect.
Contributor guide
Assessment
This issue has not been assessed yet.