How to load Kmeans model
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I use dask_ml Kmeans to traing a model (I use dask on yarn), and I just dump the model when fit finished, like the following code snippet:
```
from dask_ml.cluster import KMeans
from sklearn.externals import joblib
km = KMeans(n_clusters=500, max_iter=500)
km.fit(fea_arr)
joblib.dump(km, './kmeans_model.pkl')
```
And I get the kmeans_model.pkl file, when I load the model in python interpreter it throw some error like the following:
```
>>> from sklearn.externals import joblib
>>> joblib.load('kmeans_model.pkl')
Traceback (most recent call last):
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/comm/core.py", line 190, in connect
quiet_exceptions=EnvironmentError)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
tornado.util.TimeoutError: Timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 598, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 526, in _unpickle
obj = unpickler.load()
File "/usr/lib/python3.5/pickle.py", line 1039, in load
dispatch[key[0]](self)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 339, in load_build
Unpickler.load_build(self)
File "/usr/lib/python3.5/pickle.py", line 1510, in load_build
setstate(state)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/client.py", line 338, in __setstate__
c = get_client(address)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/worker.py", line 2577, in get_client
return Client(address, timeout=timeout)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/client.py", line 636, in __init__
self.start(timeout=timeout)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/client.py", line 759, in start
sync(self.loop, self._start, **kwargs)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/utils.py", line 277, in sync
six.reraise(*error[0])
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/six.py", line 693, in reraise
raise value
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/utils.py", line 262, in f
result[0] = yield future
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/client.py", line 847, in _start
yield self._ensure_connected(timeout=timeout)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/client.py", line 885, in _ensure_connected
connection_args=self.connection_args)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/comm/core.py", line 201, in connect
_raise(error)
File "/home/ubuntu/data/richard/py3/lib/python3.5/site-packages/distributed/comm/core.py", line 182, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://172.31.17.12:46586' after 3 s: in : ConnectionRefusedError: [Errno 111] Connection refused
```
I think the model contains some information about the dask worker and try to restore the model object in the worker, but the worker process is already exit, all the worker exit.
How can I load the model? and can I load the model in single machine like it is trained in a single machine? Thank you for you help!
Contributor guide
Assessment
This issue has not been assessed yet.