discover_jsonlines error
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I have a jsonline file and i tried to use odo in order to get the right object to start to put my jsonlines file data into my mongo with the following command:
odo.odo(odo.JSONLines("./1477858521/16706316"),"mongodb://localhost/twitter::timelines")
However, I am having the following error :
```
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in ()
----> 1 odo.odo(odo.JSONLines("./1477858521/16706316"),"mongodb://localhost/twitter::timelines")
/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/odo.py in odo(source, target, **kwargs)
89 odo.append.append - Add things onto existing things
90 """
---> 91 return into(target, source, **kwargs)
/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/multipledispatch/dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:
/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/into.py in wrapped(*args, **kwargs)
41 raise TypeError('dshape argument is not an instance of DataShape')
42 kwargs['dshape'] = dshape
---> 43 return f(*args, **kwargs)
44 return wrapped
45
/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/into.py in into_string(uri, b, dshape, **kwargs)
136 def into_string(uri, b, dshape=None, **kwargs):
137 if dshape is None:
--> 138 dshape = discover(b)
139
140 resource_ds = 0 * dshape.subshape[0] if isdimension(dshape[0]) else dshape
/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/multipledispatch/dispatcher.py in __call__(self, *args, **kwargs)
162 self._cache[types] = func
163 try:
--> 164 return func(*args, **kwargs)
165
166 except MDNotImplementedError:
/home/vagrant/.virtualenvs/venvpy3/lib/python3.4/site-packages/odo/backends/json.py in discover_jsonlines(j, n, encoding, **kwargs)
93 ds = discover(data)
94 else:
---> 95 ds = var * discover(data).subshape[0]
96 return date_to_datetime_dshape(ds)
97
AttributeError: 'Tuple' object has no attribute 'subshape'
```
After analysis, it looks like some paranthesis are missing in the following lines :
```
@discover.register(JSONLines)
def discover_jsonlines(j, n=10, encoding='utf-8', **kwargs):
with json_lines(j.path, encoding=encoding) as lines:
data = pipe(lines, filter(nonempty), map(json.loads), take(n), list)
if len(data) < n:
ds = discover(data)
else:
ds = var * discover(data).subshape[0]
return date_to_datetime_dshape(ds)
```
`
ds = var * discover(data).subshape[0]
`
should be
`
ds = (var * discover(data)).subshape[0]
`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the discover_jsonlines function in odo/backends/json.py and reproduce the reported JSONLines-to-MongoDB command. Check the failing discovery expression against the surrounding data-shape handling, then verify that discovery completes without the AttributeError and that the reported command progresses successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100