dotnet / dotnet/machinelearning

Powershell Integration - getting errors

Open
#3,669 2 comments 0 reactions 0 assignees View on GitHub
area-Core enhancement needs-author-action Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### System information

- **Windows 10, Linux (Ubuntu)**:
- **netstandard2, core 2**:
- **ML version 1.0.0-preview**:

### Issue

I'm trying to integrate ML.NET with powershell. I'm using [Sentiment Analysis sample](https://github.com/dotnet/samples/tree/master/machine-learning/tutorials/SentimentAnalysis) for testing. Sample code runs just fine as a console app, but throws errors if executed as Powershell Cmdlet. It doesn't matter if it's binary cmdlet (C#, using same exact code as console app) or pure PS. It doesn't work even if I save sample as a class library (and calling static methods via PS).

- **Shuffle input cursor reader failed with an exception**
If disabling shuffling, getting this:
- **Splitter/consolidator worker encountered exception while consuming source data**

This happens when calling fit method after trainer is added to the pipeline (I can call fit on estimator with no error). I tried Log regression and Fast Tree trainers (getting same errors)

Not sure if there is any fundamental blocker, but I think it would be extremely useful to get it work with PowerShell

### Source code / logs

```PowerShell

<# Downloading assemblies and data set

# download nuget if needed
# iwr "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe" -OutFile "nuget.exe"

nuget install Microsoft.ML -version 1.0.0-preview

mkdir bin

gci "*\lib\netstandard*\*.dll" | copy-item -Destination ".\bin"

$url = "https://raw.githubusercontent.com/lucasalexander/mlnet-samples/master/sentiment-analysis/data/yelp_labelled.txt"
Invoke-WebRequest -Uri $url -OutFile "yelp_labelled.txt"

#>

Add-Type -Path "$pwd\bin\*.dll"

$dataPath = "$pwd\yelp_labelled.txt"

$mlCOntext = [Microsoft.ML.MLContext]::new()

$columns = [System.Collections.Generic.List``1[Microsoft.ML.Data.TextLoader+Column]]::new()

$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("SentimentText", "String", 0))
$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("Label", "Boolean", 1))

$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("PredictedLabel", "Boolean", 2))
$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("Probability", "Single", 3))
$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("Score", "Single", 4))

$opt = [Microsoft.ML.Data.TextLoader+Options]::new()
$opt.Separators = "`t"
$opt.Columns = $columns
$opt.HasHeader = $false

$dataView = [Microsoft.ML.TextLoaderSaverCatalog]::LoadFromTextFile($mlCOntext.Data, $dataPath, $opt)

# preview data
# [Microsoft.ML.DebuggerExtensions]::Preview($dataView).rowview | foreach { $_.Values.Value -join " | " }

$splitDataView = $mlCOntext.Data.TrainTestSplit($dataView, 0.2)
$trainSet = $splitDataView.TrainSet
$testSet = $splitDataView.TestSet

$estimator = [Microsoft.ML.TextCatalog]::FeaturizeText($mlCOntext.Transforms.Text, "Features", "SentimentText")

$optTrain = [Microsoft.ML.Trainers.SdcaLogisticRegressionBinaryTrainer+Options]::new()
$optTrain.FeatureColumnName = "Features"
$optTrain.LabelColumnName = "Label"

# this will avoid 'Shuffle input cursor' error, but raise 'Splitter/consolidator' error
$optTrain.Shuffle = $false

$trainer = [Microsoft.ML.StandardTrainersCatalog]::SdcaLogisticRegression($mlCOntext.BinaryClassification.Trainers, $optTrain)

<# fast tree trainer, but getting Splitter/consolidator error again
$topt = [Microsoft.ML.Trainers.FastTree.FastTreeBinaryTrainer+Options]::new()
$topt.FeatureColumnName = "Features"
$topt.LabelColumnName = "Label"
$trainer = [Microsoft.ML.TreeExtensions]::FastTree($mlCOntext.BinaryClassification.Trainers, $topt)
#>

$pipe = [Microsoft.ML.LearningPipelineExtensions]::Append($estimator, $trainer, "Everything")

$model = $pipe.Fit($trainSet) # ERROR OCCURS HERE !!!

# if apply fit on estimator no error will occur and predict/evaluate block will work (with some dummy results)

# $model = $estimator.Fit($splitDataView.TrainSet)

$predict = $model.Transform($TestSet)

$mlCOntext.BinaryClassification.Evaluate($predict, "Label")
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.