dask / dask/dask-ml

LinearRegression Exception: "AttributeError: 'tuple' object has no attribute 'shape'"

Open
#770 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
In order to use dask-ml models to train on a dask DataFrame, the DataFrame must be converted to a dask array.

```python

#df is a dask DataFrame that looks something like this:
#
# flower.petal_length flower.petal_width flower.sepal_length
# npartitions=73
# float64 float64 float64
# ... ... ...
# ... ... ... ...
# ... ... ...
# ... ... ...
# Dask Name: astype, 1185 tasks

train_array = df.to_dask_array(lengths=True)
train_labels_array = df.to_dask_array(lengths=True)

# If I simply do a train_array.compute() or train_labels_array.compute() here & comment out .fit(), there are no problems

dask_model = LinearRegression()
dask_model.fit(train_array, train_labels_array)
```

When calling .fit on LinearRegression or LogisticRegression, I'm receiving the following output from the dask cluster:

```
distributed.worker - WARNING - Compute Failed
Function: subgraph_callable
args: (('rechunk-merge-ff32808be3096a08028f7c8aa2a4bae3', 66, 0))
kwargs: {}
Exception: AttributeError("'tuple' object has no attribute 'shape'",)
```

Followed by the following exception being thrown:
```
File "my_model.py", line 89, in model
dask_model.fit(train_array, train_labels_array)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask_ml/linear_model/glm.py", line 187, in fit
self._coef = algorithms._solvers[self.solver](X, y, **solver_kwargs)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask_glm/utils.py", line 17, in normalize_inputs
mean, std = da.compute(X.mean(axis=0), X.std(axis=0))
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask/base.py", line 567, in compute
results = schedule(dsk, keys, **kwargs)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/distributed/client.py", line 2676, in get
results = self.gather(packed, asynchronous=asynchronous, direct=direct)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/distributed/client.py", line 1991, in gather
asynchronous=asynchronous,
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/distributed/client.py", line 832, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/distributed/utils.py", line 340, in sync
raise exc.with_traceback(tb)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/distributed/utils.py", line 324, in f
result[0] = yield future
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/distributed/client.py", line 1850, in _gather
raise exception.with_traceback(traceback)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask/optimization.py", line 963, in __call__
return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask/core.py", line 151, in get
result = _execute_task(task, cache)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
return func(*(_execute_task(a, cache) for a in args))
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask/utils.py", line 30, in apply
return func(*args, **kwargs)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask/array/reductions.py", line 594, in mean_chunk
n = numel(x, dtype=dtype, **kwargs)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/dask/array/reductions.py", line 558, in numel
shape = x.shape
AttributeError: 'tuple' object has no attribute 'shape'
```

**What you expected to happen**:

**Minimal Complete Verifiable Example**:
Unfortunately, I'm unable to come up with a minimal example that replicates the existing behavior. The DataFrame above is going through several steps before arriving at the point of being converted to an array (being published in a cluster, having the index reset, potentially undergoing several transformations via task_graphs and delayed(func) to .compute() calls). I'm including a basic example of what's happening above to demonstrate, but when doing this in a dask LocalCluster separate from the other environment I don't see the same issue.

**Environment**:

- Dask version: 2.10.1
- Dask-ml version: 1.7.0
- dask-glm version: 0.2.0
- sklearn version: 0.23.2
- Python version: 3.6.8
- Operating System: Centos7
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Research direction

Start with the supplied LinearRegression.fit call and traceback, then inspect dask_ml/linear_model/glm.py, dask_glm/utils.py, and the dask array reductions path named in the stack trace. Reproduce the failure using the reported Dask, dask-ml, dask-glm, scikit-learn, and Python versions; done means the tuple/shape failure is explained and a regression test covers the affected fit path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.