dotnet / dotnet/machinelearning

IMonitor.ReportRunningTrial reports incorrect pipeline

Open
#6,425 4 comments 0 reactions 0 assignees View on GitHub
area-AutoML
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**System Information (please complete the following information):**
- OS & Version: Windows 11 [Version 10.0.22621.675]
- ML.NET Version: ML.NT v2.0.0-preview.22525.3
- .NET Version: .NET 6.0.9

**Describe the bug**
I created a custom class that implements the `Microsoft.ML.AutoML.IMonitor` to monitor my training. It's `ReportRunningTrial` method has the following simple implementation:
`_logger.LogInformation($"{"Running".PadLeft(9)} Trial #{setting.TrialId.ToString().PadLeft(4)} - Pipeline: {_pipeline.ToString(setting.Parameter).PadRight(70)}");`

The other, `ReportBestTrial`, `ReportCompletedTrial` and `ReportFailTrial` methods look very similar.

When I run my training it looks like that the `ReportRunningTrial` method's pipeline object wasn't updated, so it doesn't show the pipeline of the current trial.

**To Reproduce**
Steps to reproduce the behavior:
1. Create [a custom class](https://github.com/andrasfuchs/BioBalanceDetector/blob/master/Software/Source/BBDProto08/BBD.BodyMonitor/BBD.BodyMonitor.API/Models/MLContextLoggerMonitor.cs) that implements IMonitor
2. Create an ML.NET experiment, and use that monitor class, like this:
```
var experiment = mlContext.Auto().CreateExperiment();

experiment
.SetPipeline(pipeline)
.SetTrainingTimeInSeconds(trainingTimeInSeconds)
.SetRegressionMetric(RegressionMetric.RSquared, labelColumn: "Label")
.SetDataset(trainTestData.TrainSet, trainTestData.TestSet)
.SetMonitor(monitor)
.SetMaximumMemoryUsageInMegaByte(20 * 1024);
```
3. Make sure that you allow more than one type of (regression) algorithms
4. Run the experiment
5. Check the output: you will see that the `settings.TrialId` property values look good, but the `_pipeline.ToString(setting.Parameter)` returns an invalid value in the `ReportRunningTrial` method.

**Expected behavior**
The `_pipeline.ToString(setting.Parameter)` should return the just-started trial's pipeline.

**Screenshots, Code, Sample Projects**
![image](https://user-images.githubusercontent.com/910321/199558208-64c383b5-0c10-4fc2-a5bd-79ddf07f7275.png)

**Additional context**
You can see the problem on the screenshot:
Running Trial 0's pipeline looks like it was `FastTreeRegression`, but in reality it's `FastForestRegression`.
Running Trial 1's pipeline looks like it was `FastForestRegression`, but it's `FastTreeRegression`.
Running Trial 2's pipeline looks like it was `FastForestRegression`, but it's `LightGbmRegression`.
etc.

Edit: After testing a little more I'm not sure anymore which methods of the `IMonitor` interface report the pipeline wrong, it is possible that `ReportRunningTrial` is the correct one and the others are incorrect.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.