dotnet / dotnet/machinelearning-samples

Porting samples to PowerShell

Aperta
#407 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
follow-up
Lingua principale
PowerShell
Stelle
4.7k
Fork
2.7k
Merge medio
2g 22h
PR unite (30g)
1

Descrizione

I'm experimenting with translating some samples from C# to PowerShell (Sentiment Analysis for now). PS doesn't support (work well) with some C# features (like extension methods, attributes, etc), but it doesn't feel impossible to port it. At this point I'm struggling with executing fit method after appending Trainer to Estimator. When I follow C# sample, I'm getting "Shuffle input cursor reader" error. If I change trainer option to disable shuffling I'm getting "Splitter/consolidator worker" error. Interestingly, if I apply fit directly on estimator I'm not getting errors, and run prediction/evaluation with some dummy results. Getting same error on PS 5.1 and core 6.2. C# sample works fine on the same machine (with dotnet run).
So at least I want to clarify if there is any blocker for PS to interact with ml.net I think it would be extremely useful to port it to PS, so even non-developers might use it. Below is my code (it will export data set and libraries if needed)
```powershell
<# Downloading assemblies and data set

# download nuget if needed
# iwr "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe" -OutFile "nuget.exe"

nuget install Microsoft.ML -version 1.0.0-preview

mkdir bin

gci "*\lib\netstandard*\*.dll" | copy-item -Destination ".\bin"

$url = "https://raw.githubusercontent.com/lucasalexander/mlnet-samples/master/sentiment-analysis/data/yelp_labelled.txt"
Invoke-WebRequest -Uri $url -OutFile "yelp_labelled.txt"

#>

Add-Type -Path "$pwd\bin\*.dll"

$dataPath = "$pwd\yelp_labelled.txt"

$mlCOntext = [Microsoft.ML.MLContext]::new()

$columns = [System.Collections.Generic.List``1[Microsoft.ML.Data.TextLoader+Column]]::new()

$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("SentimentText", "String", 0))
$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("Label", "Boolean", 1))

$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("PredictedLabel", "Boolean", 2))
$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("Probability", "Single", 3))
$columns.Add([Microsoft.ML.Data.TextLoader+Column]::new("Score", "Single", 4))

$opt = [Microsoft.ML.Data.TextLoader+Options]::new()
$opt.Separators = "`t"
$opt.Columns = $columns
$opt.HasHeader = $false

$dataView = [Microsoft.ML.TextLoaderSaverCatalog]::LoadFromTextFile($mlCOntext.Data, $dataPath, $opt)

# preview data
# [Microsoft.ML.DebuggerExtensions]::Preview($dataView).rowview | foreach { $_.Values.Value -join " | " }

$splitDataView = $mlCOntext.Data.TrainTestSplit($dataView, 0.2)
$trainSet = $splitDataView.TrainSet
$testSet = $splitDataView.TestSet

$estimator = [Microsoft.ML.TextCatalog]::FeaturizeText($mlCOntext.Transforms.Text, "Features", "SentimentText")

$optTrain = [Microsoft.ML.Trainers.SdcaLogisticRegressionBinaryTrainer+Options]::new()
$optTrain.FeatureColumnName = "Features"
$optTrain.LabelColumnName = "Label"

# this will avoid 'Shuffle input cursor' error, but raise 'Splitter/consolidator' error
#$optTrain.Shuffle = $false

$trainer = [Microsoft.ML.StandardTrainersCatalog]::SdcaLogisticRegression($mlCOntext.BinaryClassification.Trainers, $optTrain)

$pipe = [Microsoft.ML.LearningPipelineExtensions]::Append($estimator, $trainer, "Everything")

$model = $pipe.Fit($trainSet) # GETTING ERROR HERE !

# if apply fit on estimator no error will occur and predict/evaluate block will work (with some dummy results)

# $model = $estimator.Fit($splitDataView.TrainSet)

$predict = $model.Transform($TestSet)

$mlCOntext.BinaryClassification.Evaluate($predict, "Label")
```

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia eseguendo la riproduzione PowerShell fornita tramite $pipe.Fit($trainSet), quindi confrontala con l'esempio funzionante di analisi del sentiment in C# sulla stessa macchina. Traccia gli errori segnalati “Shuffle input cursor reader” e “Splitter/consolidator worker”; il lavoro è completato quando viene documentato se l'interazione tra PowerShell e ML.NET è bloccata e viene fornita una riproduzione confermata funzionante o con un fallimento minimo.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
csharp, powershell
Ambito
machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
28/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.