dotnet / dotnet/machinelearning-modelbuilder

Remove '\n' and '\r' from datasets

Offen
#1,008 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Code Example? Priority:2 Reported by: Customer
Vorherrschende Sprache
Dockerfile
Sterne
285
Forks
66
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

System Information:
- Model Builder Version: 16.7.3
- Visual Studio Version: 16.1.0

The ML Builder fails at training part with these messages:
`
2020-09-21 16:28:34.7573 DEBUG Set log file path to C:\Users\Admin\AppData\Local\Temp\MLVSTools\logs\9a23c49b-556a-484f-bb0f-63101f9af44f.txt (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:28:44.0690 DEBUG C:\Users\Admin\Documents\datasets\spam dataset\spamFixed.csv (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:28:58.7341 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:28:59.6822 WARN GPU Service not found. Falling back to CPU AutoML Service. (Microsoft.ML.ModelBuilder.Utils.Logger.Warn)
2020-09-21 16:29:01.8903 DEBUG Unable to split the file provided into multiple, consistent columns.
at Microsoft.ML.AutoML.ColumnInferenceApi.InferSplit(MLContext context, TextFileSample sample, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse)
at Microsoft.ML.AutoML.ColumnInferenceApi.InferColumns(MLContext context, String path, ColumnInformation columnInfo, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse, Boolean trimWhitespace, Boolean groupColumns, Boolean hasHeader)
at Microsoft.ML.ModelBuilder.AutoMLEngine.InferColumns(MLContext context, AutoMLServiceParamater config, ColumnInformation columnInformation)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 112 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:35:04.3941 DEBUG C:\Users\Admin\Documents\datasets\spam dataset\spamFixed.csv (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:35:13.1413 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:35:13.1783 DEBUG Unable to split the file provided into multiple, consistent columns.
at Microsoft.ML.AutoML.ColumnInferenceApi.InferSplit(MLContext context, TextFileSample sample, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse)
at Microsoft.ML.AutoML.ColumnInferenceApi.InferColumns(MLContext context, String path, ColumnInformation columnInfo, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse, Boolean trimWhitespace, Boolean groupColumns, Boolean hasHeader)
at Microsoft.ML.ModelBuilder.AutoMLEngine.InferColumns(MLContext context, AutoMLServiceParamater config, ColumnInformation columnInformation)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 112 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:20.4254 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:20.4254 DEBUG Disposing AutoMLService Client (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:20.4254 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:20.4454 DEBUG The process cannot access the file 'C:\Users\Admin\Documents\datasets\spam dataset\spamFixed.csv' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at Microsoft.ML.AutoML.TextFileSample.CreateFromFullFile(String path)
at Microsoft.ML.AutoML.ColumnInferenceApi.InferColumns(MLContext context, String path, ColumnInformation columnInfo, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse, Boolean trimWhitespace, Boolean groupColumns, Boolean hasHeader)
at Microsoft.ML.ModelBuilder.AutoMLEngine.InferColumns(MLContext context, AutoMLServiceParamater config, ColumnInformation columnInformation)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 112 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:24.3317 DEBUG Open Log FileC:\Users\Admin\AppData\Local\Temp\MLVSTools\logs\9a23c49b-556a-484f-bb0f-63101f9af44f.txt (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:48.1415 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:48.1455 DEBUG Disposing AutoMLService Client (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:48.1455 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:48.1725 DEBUG Unable to split the file provided into multiple, consistent columns.
at Microsoft.ML.AutoML.ColumnInferenceApi.InferSplit(MLContext context, TextFileSample sample, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse)
at Microsoft.ML.AutoML.ColumnInferenceApi.InferColumns(MLContext context, String path, ColumnInformation columnInfo, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse, Boolean trimWhitespace, Boolean groupColumns, Boolean hasHeader)
at Microsoft.ML.ModelBuilder.AutoMLEngine.InferColumns(MLContext context, AutoMLServiceParamater config, ColumnInformation columnInformation)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 112 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:36:49.5316 DEBUG Open Log FileC:\Users\Admin\AppData\Local\Temp\MLVSTools\logs\9a23c49b-556a-484f-bb0f-63101f9af44f.txt (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:04.9753 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:04.9753 DEBUG Disposing AutoMLService Client (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:05.1113 DEBUG Disposing AutoMLService Client (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:05.1113 DEBUG Disposing AutoMLService Client (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:05.1113 DEBUG Disposing AutoMLService Client (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:15.6691 DEBUG C:\Users\Admin\Documents\datasets\spam dataset\spamFixed.csv (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:26.4699 DEBUG Disposing TrainSession (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:26.4929 DEBUG Unable to split the file provided into multiple, consistent columns.
at Microsoft.ML.AutoML.ColumnInferenceApi.InferSplit(MLContext context, TextFileSample sample, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse)
at Microsoft.ML.AutoML.ColumnInferenceApi.InferColumns(MLContext context, String path, ColumnInformation columnInfo, Nullable`1 separatorChar, Nullable`1 allowQuotedStrings, Nullable`1 supportSparse, Boolean trimWhitespace, Boolean groupColumns, Boolean hasHeader)
at Microsoft.ML.ModelBuilder.AutoMLEngine.InferColumns(MLContext context, AutoMLServiceParamater config, ColumnInformation columnInformation)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 112 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-09-21 16:37:27.6030 DEBUG Open Log FileC:\Users\Admin\AppData\Local\Temp\MLVSTools\logs\9a23c49b-556a-484f-bb0f-63101f9af44f.txt (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
`

Steps to reproduce the behavior:
Train a Text Classifier using this dataset: https://www.kaggle.com/uciml/sms-spam-collection-dataset

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Reproduzieren Sie den Fehler mit dem verknüpften Kaggle-SMS-Spam-Dataset und beginnen Sie dann bei AutoMLService/AutoMLEngineService/AutoMLEngine.cs, Zeile 112, und folgen Sie dem Aufruf zur Spaltenerkennung in ML.NETs InferColumns-Pfad. Bestätigen Sie, dass Wagenrücklauf- und Zeilenumbruchzeichen im Dataset keine Inkonsistente-Spalten-Fehler mehr verursachen und dass das Training des Textklassifizierers abgeschlossen wird.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
csharp, machine-learning
Bereich
data, machine-learning
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.