dotnet / dotnet/machinelearning
[ML.net cli] Resume a cli training after a crash
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
Hello, is there a way to resume an ML.net cli training to where it was before a crash?
I have a lot of data in the folder C:\Users\wwww\AppData\Local\Temp\AutoML-NNI\Experiment-9K67B4 but I do not know how to make mlnet start from there.
Detail:
I used the cli, ie "mlnet classicfiaction...."
I trained for a few days, but I made a mistake which used a lot of memory on my computer, which stopped the mlnet process. I would like to start mlnet to where it left so it can continue from there
Thanks
w
2022-08-10 15:03:24.3091 DEBUG System.InvalidOperationException: Event we were waiting on was subject to an exception
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Array.Resize[T](T[]& array, Int32 newSize)
at Microsoft.ML.Internal.Utilities.ArrayUtils.EnsureSize[T](T[]& array, Int32 min, Int32 max, Boolean keepOld, Boolean& resized)
at Microsoft.ML.Data.CacheDataView.ColumnCache.ImplOne`1.CacheCurrent()
at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)
Contributor guide
Assessment
This issue has not been assessed yet.