blaze / blaze/odo

discover_jsonlines error

Open
#493 2 comments 1 reaction 0 assignees View on GitHub
bug json mongo
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have a jsonline file and i tried to use odo in order to get the right object to start to put my jsonlines file data into my mongo with the following command:

odo.odo(odo.JSONLines("./1477858521/16706316"),"mongodb://localhost/twitter::timelines")

However, I am having the following error :
```
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in ()
----> 1 odo.odo(odo.JSONLines("./1477858521/16706316"),"mongodb://localhost/twitter::timelines")

/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/odo.py in odo(source, target, **kwargs)
89 odo.append.append - Add things onto existing things
90 """
---> 91 return into(target, source, **kwargs)

/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/multipledispatch/dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/into.py in wrapped(*args, **kwargs)
41 raise TypeError('dshape argument is not an instance of DataShape')
42 kwargs['dshape'] = dshape
---> 43 return f(*args, **kwargs)
44 return wrapped
45

/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/into.py in into_string(uri, b, dshape, **kwargs)
136 def into_string(uri, b, dshape=None, **kwargs):
137 if dshape is None:
--> 138 dshape = discover(b)
139
140 resource_ds = 0 * dshape.subshape[0] if isdimension(dshape[0]) else dshape

/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/multipledispatch/dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:

/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/backends/json.py in discover_jsonlines(j, n, encoding, **kwargs)
93 ds = discover(data)
94 else:
---> 95 ds = var * discover(data).subshape[0]
96 return date_to_datetime_dshape(ds)
97

AttributeError: 'Tuple' object has no attribute 'subshape'
```

After analysis, it looks like some paranthesis are missing in the following lines :

```
@discover.register(JSONLines)
def discover_jsonlines(j, n=10, encoding='utf-8', **kwargs):
with json_lines(j.path, encoding=encoding) as lines:
data = pipe(lines, filter(nonempty), map(json.loads), take(n), list)

if len(data) < n:
ds = discover(data)
else:
ds = var * discover(data).subshape[0]
return date_to_datetime_dshape(ds)
```

`
ds = var * discover(data).subshape[0]
`
should be

`
ds = (var * discover(data)).subshape[0]
`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the discover_jsonlines function in odo/backends/json.py and reproduce the reported JSONLines-to-MongoDB command. Check the failing discovery expression against the surrounding data-shape handling, then verify that discovery completes without the AttributeError and that the reported command progresses successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.