blaze / blaze/blaze

Group by with nunique reduction on bcolz throws NotImplementedError

Open
#1,620 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.2k
Forks
389
PR merge metrics
No merged PRs in 30d

Description

I converted some 2M json records to a bcolz file using odo in the hopes of experimenting with blaze's query features. But unfortunately I am still new to the blaze world so I don't know if this is an error or to be expected for a bcolz backend. The `nunique` function works fine on its own but not when placed within a `by` expression.

If it helps, I can reproduce this with fake data but I thought I'd put this out there to know for sure this is either unexpected or expected behavior.

I am using Python 3.5.2, blaze 0.11.3, bcolz 1.1.0.

```
>>> from blaze import *
>>> d = data('data.bcolz')
>>> nunique(d.username)
932
>>> by(d.app_ver, n=count(d.username))
app_ver n
0 1_0 18109
1 1_0.0 32594
2 1_0.1 11931
3 1_0.4 64228
4 1_0.5 2167670
5 1_1.0 102524
6 1_1.1 36819
7 1_2 21000
8 1_2.0 18
9 1_2.2 37139
>>> by(d.app_ver, n=nunique(d.username))
Traceback (most recent call last):
....
NotImplementedError: Don't know how to compute:
type(expr):
expr: by(_.app_ver, n=nunique(_.birth_date)).head(11)
data: {<`_` symbol; dshape='2830325 * { app_ver: string[10, 'U32'], username...'>: .c object at 0x7fd5804e28e8>}
```

I am using an Anaconda environment and the blaze channel to install the current release:

```{yaml}
name: blaze
channels:
- blaze
dependencies:
- python=3.5.2
- blaze=0.11.3
- pip
- pip:
- bcolz
````

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the documented Python 3.5.2, blaze 0.11.3, and bcolz 1.1.0 example with data('data.bcolz'), comparing nunique(d.username) with by(d.app_ver, n=nunique(d.username)). Trace the by and nunique entry points into the bcolz backend. Done means grouped nunique has a defined, tested result or the backend's unsupported behavior is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.