dotnet / dotnet/machinelearning
Fit() method should be optimized when used with DatabaseLoader or IEnumerable (with sqldatareader)
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
**Is your feature request related to a problem? Please describe.**
Fit() method is not optimized when used with databaseloader or ienumerable(sqldatareader->yield return) and is causing delays in processing.
The training data that our ML.Net(v 1.7) project refers to resides in Sql Server database
I do not want to load the entire data in memory for fitting
In order to fetch this data while training I have tried 2 ways:
1. using Databaseloader which has a stored procedure as command text
2. using IEnumerable which calls sqldatareader(with yield return)(this also calls a stored procedure)
and then creating IDataView using LoadFromEnumerable()
The training data has 6 columns which are transformed as follows:
FeaturizeText is being called for 4 columns
Onehotencoding is being called for 2 columns
Concatenation of all above transformed columns
**Issue**
In both above ways of fetching:
**The underlying stored procedure is getting called 15 times during each “Fit” call**
I observed that the calls are due to the transforms being applied
If I reduce the number of transforms, the calls to stored procedure reduce accordingly
This causes the Fit() method to take considerable amount of time
**Describe the solution you'd like**
1. Ideally the data should be fetched just once and all preprocessing done on the fetched data
2.Please share a sample that uses IEnumerable with sqldatareader, as this is my preferred approach
I would compare it with my implementation
**Describe alternatives you've considered**
None
**Additional context**
None
Contributor guide
Assessment
This issue has not been assessed yet.