dotnet / dotnet/machinelearning-modelbuilder
Suggestion: Improve performance by running experiments in parallel by monitoring available system resources
- Dominant language
- Dockerfile
- Stars
- 285
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Some algorithms utilize CPU fully with good multi-threading performance. Others seem to only utilize single core, yet other algorithms need large amount of system memory and are mostly waiting for paging file access (as it often does not fit in RAM).
**Describe the solution you'd like**
Would it be possible to mix and match algorithms based on their resource consumption profile?
For example, run a single-core algorithm's experiment while running a multi-threading algorithm on the remaining cores.
Or, potentially ran an algorithm with heavy RAM usage along low RAM usage algorithm.
Even better would be if the AutoML library would monitor what resources are available and spin up to new experiments to utilize more of the resources available.
**Describe alternatives you've considered**
Currently, two or more different experiment can be run simultaneously manually (by adding two .mbconfig files and running them manually). The problem with this solution is that two RAM-intensive experiments may run at same time, consuming enormous amount of RAM / system memory, leading often to a crash and loosing the progress. (Single experiment with 2 Gb dataset could consume over 500Gb of system memory https://github.com/dotnet/machinelearning-modelbuilder/issues/1531 )
Alternative could be a feature to help developers reduce the training data, such as select a sub-sample to run experiments faster with smaller datasets. Only at stage 2, best performed models could be run with full data for comparison.
The reason why this should be considered is that many of the production AI models are based on moderate amounts of data, and the training may take at least a few days even with high-end devices with full data set to get to good results. In worst case, the time assigned may run out of time before a suitable model was encountered.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.