Blaze server choking on data
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 389
- PR merge metrics
- No merged PRs in 30d
Description
I'm working with very nested directory structure of ~ 10,000 json files. On the server I do:
``` python
In [3]:
from blaze import resource, Server
In [4]:
import os
def get_resources(resource_path):
resources = {}
for i, (root, dirs, files) in enumerate(os.walk(resource_path)):
for file in files:
path = os.path.join(root, file)
resources[path.replace('/', ' ')] = resource(path)
return resources
r = get_resources('lastfm_subset/A')
server = Server(r)
In [5]:
len(r)
Out[5]:
7087
In [*]:
server.run()
```
And on the client I try to make a data object from the blaze server URI. It returns some html with a 500 Internal server error. Back on the server I get:
```
ERROR:blaze.server.server:Exception on /datashape [GET]
Traceback (most recent call last):
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/blaze/server/server.py", line 135, in dataset
return str(discover(_get_data()))
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in discover
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/odo/backends/json.py", line 79, in discover_json
ds = discover(data)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in discover
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 185, in discover
for column in columns]
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 185, in
for column in columns]
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 318, in f
result = func(inp)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 272, in unite_base
base = lowest_common_dshape(good_dshapes)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 253, in lowest_common_dshape
raise ValueError("Not all dshapes are known. Extend edges.")
ValueError: Not all dshapes are known. Extend edges.
```
So is some of my data bad? It would be nice if blaze told me which files are bad so I could omit them.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.