dotnet / dotnet/machinelearning

Fit() method should be optimized when used with DatabaseLoader or IEnumerable (with sqldatareader)

Open
#6,050 6 comments 0 reactions 0 assignees View on GitHub
area-Core tenet-performance
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**
Fit() method is not optimized when used with databaseloader or ienumerable(sqldatareader->yield return) and is causing delays in processing.

The training data that our ML.Net(v 1.7) project refers to resides in Sql Server database

I do not want to load the entire data in memory for fitting
In order to fetch this data while training I have tried 2 ways:
1. using Databaseloader which has a stored procedure as command text
2. using IEnumerable which calls sqldatareader(with yield return)(this also calls a stored procedure)
and then creating IDataView using LoadFromEnumerable()

The training data has 6 columns which are transformed as follows:
FeaturizeText is being called for 4 columns
Onehotencoding is being called for 2 columns
Concatenation of all above transformed columns

**Issue**
In both above ways of fetching:
**The underlying stored procedure is getting called 15 times during each “Fit” call**
I observed that the calls are due to the transforms being applied
If I reduce the number of transforms, the calls to stored procedure reduce accordingly
This causes the Fit() method to take considerable amount of time

**Describe the solution you'd like**
1. Ideally the data should be fetched just once and all preprocessing done on the fetched data
2.Please share a sample that uses IEnumerable with sqldatareader, as this is my preferred approach
I would compare it with my implementation

**Describe alternatives you've considered**
None

**Additional context**
None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.