dotnet / dotnet/machinelearning
AutoML 2.0: Distributes learning / potential "hack" using checkpoints CSV on a network drive?
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
I could not find docs or plans about distributed learning. If that is correct, any ideas for how it could be implemented or links to common approaches?
AutoML 2.0 seems to have a nice checkpoint CSV file.
Would it be possible to make a "hack" using the checkpoint file? Would there a tuner that would allow me to use it for distributed tuning? Or, perhaps setting different MLContext seed on each machine would do it?
I am thinking something like this might work with minimal source editing:
1. Checkpoint file to a network drive
2. Make each experiment run only 1 trial, then restart (so the current status gets read) - or edit source to re-read the checkpoint file
3. Multiple clients would run simultaneously
Potential problems I could expect:
1. Some type of randomity is probably needed from tuner, otherwise all clients might run the trial on same parameters
2. Probably needs to add locks for saving to CSV
Contributor guide
Assessment
This issue has not been assessed yet.