blaze / blaze/blaze

Blaze server choking on data

Open
#1,111 5 comments 0 reactions 0 assignees View on GitHub
bug dependency
Dominant language
Python
Stars
3.2k
Forks
389
PR merge metrics
No merged PRs in 30d

Description

I'm working with very nested directory structure of ~ 10,000 json files. On the server I do:

``` python
In [3]:

from blaze import resource, Server
In [4]:

import os
def get_resources(resource_path):
resources = {}
for i, (root, dirs, files) in enumerate(os.walk(resource_path)):
for file in files:
path = os.path.join(root, file)
resources[path.replace('/', ' ')] = resource(path)
return resources

r = get_resources('lastfm_subset/A')
server = Server(r)
In [5]:

len(r)
Out[5]:
7087
In [*]:

server.run()
```

And on the client I try to make a data object from the blaze server URI. It returns some html with a 500 Internal server error. Back on the server I get:

```
ERROR:blaze.server.server:Exception on /datashape [GET]
Traceback (most recent call last):
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/blaze/server/server.py", line 135, in dataset
return str(discover(_get_data()))
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in discover
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/odo/backends/json.py", line 79, in discover_json
ds = discover(data)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in discover
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 342, in
return Record([[k, discover(d[k])] for k in sorted(d)])
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/multipledispatch/dispatcher.py", line 163, in __call__
return func(*args, **kwargs)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 185, in discover
for column in columns]
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 185, in
for column in columns]
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 318, in f
result = func(inp)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 272, in unite_base
base = lowest_common_dshape(good_dshapes)
File "/home/blake/.miniconda/envs/blaze3/lib/python3.4/site-packages/datashape/discovery.py", line 253, in lowest_common_dshape
raise ValueError("Not all dshapes are known. Extend edges.")
ValueError: Not all dshapes are known. Extend edges.
```

So is some of my data bad? It would be nice if blaze told me which files are bad so I could omit them.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.