NoneType has no attribute 'measure'
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to push a SQLAlchemy query into a DataFrame.
I've set up the session and set up a query (which works just fine).
Since I can't convert the myquery object directly into a dataframe, I thought I'd try the list object returned by myquery.all().
I get the following exception when I try it:
```
mydf = odo.odo(myquery.all(), DataFrame)
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/odo/odo.py", line 90, in odo
return into(target, source, **kwargs)
File "/usr/local/lib/python2.7/site-packages/multipledispatch/dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/odo/into.py", line 25, in into_type
return convert(a, b, dshape=dshape, **kwargs)
File "/usr/local/lib/python2.7/site-packages/odo/core.py", line 30, in __call__
return _transform(self.graph, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/odo/core.py", line 46, in _transform
x = f(x, excluded_edges=excluded_edges, **kwargs)
File "/usr/local/lib/python2.7/site-packages/odo/convert.py", line 170, in list_to_numpy
return np.array(seq, dtype=dshape_to_numpy(dshape))
File "/usr/local/lib/python2.7/site-packages/odo/numpy_dtype.py", line 60, in dshape_to_numpy
return unit_to_dtype(ds)
File "/usr/local/lib/python2.7/site-packages/odo/numpy_dtype.py", line 32, in unit_to_dtype
return to_numpy_dtype(ds)
File "/usr/local/lib/python2.7/site-packages/datashape/coretypes.py", line 1139, in to_numpy_dtype
return to_numpy(ds.measure)[1]
AttributeError: 'NoneType' object has no attribute 'measure'
```
There is a lot of data in my result, however I'm sure some of the columns have null values here an there. I'm not sure why ds is ending up as 'None' without spending a lot of time in the odo code. Is it because there is a null value somewhere that it isn't expecting one? Maybe it will be obvious to someone closer to the code here. I'm wondering if something should catch this before it throws an exception.
I would LOVE it if I could simply call: odo.odo(myquery, DataFrame).
odo is pretty awesome and has come in handy a few times now. Keep up the great work!
Note:
The pandas.read_sql(myquery.statement, mysession.bind) function (as seen in several examples out there on the web) does not work for me:
(1) It appears to create a new session to run the query - so it does not inherit previously created session variables (such as "search_path"). It also isn't going to lend itself to use in a Flask application where I'm sharing the session across threads. It also won't work within a larger transaction.
(2) myquery.statement does not include parameters, so it is fine for simple queries, but does not expand property for general querying.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the SQLAlchemy query-to-DataFrame conversion using the traceback's odo/convert.py and odo/numpy_dtype.py paths, including results with null values. Trace how the dshape reaches list_to_numpy and compare that with the requested direct query conversion. Done means the reported query no longer raises this exception and its resulting DataFrame behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, sqlalchemy
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100