dotnet / dotnet/machinelearning

AggregateException/InvalidOperationException when training from IEnumerable backed by EF Core Context

Open
#2,159 17 comments 0 reactions 0 assignees View on GitHub
area-Core enhancement Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### System information
- .Net Core 2.2, EF Core 2.2.1, ML.NET 0.9

### Issue

- Read data from SQL DB via EF Core as IEnumerable, pass to pipeline via streaming IDataView
- Get InvalidOperationException from Microsoft.ML.Transforms.RowShufflingTransformer.Cursor.LoopProducerWorker()
- It worked fine in ML.NET 0.8. I can work-around by forcing the trainer to be single-threaded but this makes training in 0.9 much slower than in 0.8 (on my machine).

The EF Core problem could be addressed by allowing multiple dbcontexts to be provided but this is likely to be an issue with any non thread-safe IDataView source...

### Source code / logs

Source at: https://github.com/endintiers/Unearth.Demo.ML.FromDB shows this working in 0.8, failing in 0.9 and a work-around for 0.9 (single-threading the training).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.