dask / dask/dask-ml

_partial.fit modifies estimator inplace

Open
#277 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

Questions from @mrocklin in https://github.com/dask/dask-ml/pull/275#issuecomment-402269422

> Shouldn't we be cloning the model here before calling partial fit? Otherwise we're mutating the input.
> What if we have to rerun this task because the worker that the result was on failed?

I'm not sure what happens when the worker fails :)

Let's say that

- Worker A completed `partial_fit` on the first block of data
- Worker B fails on the second block of data.

IIRC, when a worker fails during computation, the scheduler will mark the task as suspicions and reschedule the task on another worker. Let's say it's scheduled on worker C for whatever reason.

Worker C asks worker A for `fit--0`. I think everything is OK. The scheduler should always have a correct understanding of who has the latest successful fit call.

Does that sound right? Am I missing scenarios where we do something wrong?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.