dotnet / dotnet/machinelearning

AutoML 2.0: Distributes learning / potential "hack" using checkpoints CSV on a network drive?

Open
#6,648 2 comments 0 reactions 1 assignee Claimed by @LittleLittleCloud View on GitHub
area-AutoML enhancement needs-further-triage untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

I could not find docs or plans about distributed learning. If that is correct, any ideas for how it could be implemented or links to common approaches?

AutoML 2.0 seems to have a nice checkpoint CSV file.
Would it be possible to make a "hack" using the checkpoint file? Would there a tuner that would allow me to use it for distributed tuning? Or, perhaps setting different MLContext seed on each machine would do it?

I am thinking something like this might work with minimal source editing:
1. Checkpoint file to a network drive
2. Make each experiment run only 1 trial, then restart (so the current status gets read) - or edit source to re-read the checkpoint file
3. Multiple clients would run simultaneously

Potential problems I could expect:
1. Some type of randomity is probably needed from tuner, otherwise all clients might run the trial on same parameters
2. Probably needs to add locks for saving to CSV

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.