blaze / blaze/odo

Odo discover chokes on mongoDB DBRef types

Open
#435 3 comments 0 reactions 0 assignees View on GitHub
backend bug discovery mongo
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Odo is choking on a mongoDB DBRef types
not sure if this is by design but I wanted to make you aware of it.
The following code shows the issue:
_(connection actuals are hidden and printed uri changed to show masked actuals)_

``` python
from blaze import odo
from odo import resource, discover
collections = []
mongo_template = u"mongodb://{0}:{1}@{2}:{3}/{4}::{5}"
db_uri_template = mongo_template.split("::")[0]
db_uri = db_uri_template.format(username,password,host,port,db)# defined elsewhere
collections = resource(db_uri).collection_names()
uri=mongo_template.format(username,password,host,port,db,collections[6])
print('URI:')
print(uri)
print()
entity_state=resource(uri)
print('mongo record:')
print(entity_state.find_one())
ds=discover(entity_state)
```

yields the following output:

``` python
URI:
mongodb://user:pass@host:27017/db_name::entityState

mongo record:
{
'values': [],
'entity': DBRef('entityLifecycle', ObjectId('5525dd21a4a4b2a53ea03b53')),
'entityType': 'ORG',
'name': 'HistoricalValidation',
'_id': ObjectId('5525dd22a4a4b2a53ea03b61'),
'lastModifiedDate': datetime.datetime(2015, 4, 9, 2, 0, 2, 17000),
'createdDate': datetime.datetime(2015, 4, 9, 2, 0, 2, 17000),
'EntityId': 'e6bdf2cc-10ca-4107-9435-3b34162c8fff',
'timestamp': datetime.datetime(2015, 4, 9, 2, 0, 0, 1000)
}

---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
in ()
8 print('mongo record:')
9 print(entity_state.find_one())
---> 10 ds=discover(entity_state)

C:\Anaconda3\envs\oz\lib\site-packages\multipledispatch\dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

C:\Anaconda3\envs\oz\lib\site-packages\odo\backends\mongo.py in discover_pymongo_collection(coll, n)
26 del item[col]
27
---> 28 ds = discover(items)
29
30 if isdimension(ds[0]):

C:\Anaconda3\envs\oz\lib\site-packages\multipledispatch\dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

C:\Anaconda3\envs\oz\lib\site-packages\datashape\discovery.py in discover(seq)
220 try:
221 types = [unite([discover(data) for data in column]).subshape[0]
--> 222 for column in columns]
223 return len(seq) * Record(list(zip(keys, types)))
224 except AttributeError:

C:\Anaconda3\envs\oz\lib\site-packages\datashape\discovery.py in (.0)
220 try:
221 types = [unite([discover(data) for data in column]).subshape[0]
--> 222 for column in columns]
223 return len(seq) * Record(list(zip(keys, types)))
224 except AttributeError:

C:\Anaconda3\envs\oz\lib\site-packages\datashape\discovery.py in (.0)
219 columns = [[item.get(key) for item in seq] for key in keys]
220 try:
--> 221 types = [unite([discover(data) for data in column]).subshape[0]
222 for column in columns]
223 return len(seq) * Record(list(zip(keys, types)))

C:\Anaconda3\envs\oz\lib\site-packages\multipledispatch\dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

C:\Anaconda3\envs\oz\lib\site-packages\datashape\discovery.py in discover(obj, **kwargs)
64 )
65 return from_numpy(obj.shape, obj.dtype)
---> 66 raise NotImplementedError("Don't know how to discover type %r" % type_name)
67
68

NotImplementedError: Don't know how to discover type 'DBRef'

```

I'm using the following semi-relevant packages:

```
# packages in environment at C:\Anaconda3\envs\oz:
#
blaze 0.9.1 py35_0
datashape 0.5.1 py35_0
multipledispatch 0.4.8 py35_0
numpy 1.10.4 py35_0
odo 0.4.2 py35_0
pymongo 3.0.3 py35_0
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at odo/backends/mongo.py, where discover_pymongo_collection passes collection items to discovery, then inspect datashape/discovery.py and the reported DBRef failure. Reproduce with a MongoDB collection containing a DBRef and verify that discover completes without the NotImplementedError.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.