Support `group by` replacements for dimensions
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 203
Description
**Is your feature request related to a problem? Please describe.**
This feature request does relate to a problem. I am experiencing inefficient querying against my database when grouping by a text dimension that contains geojson data.
**Describe the solution you'd like**
Dimensions should take a new field that allows users to supply sql that replaces the primary sql for that dimension when used in the group by and order by sections of a query.
Take this cube for example:
```yaml
cubes:
- name: geographic_area_shapes
sql_table: '`ch_geography_silver`.`geographic_area_shapes`'
dimensions:
- name: geographic_area_key
sql: geographic_area_key
type: string
meta:
cube:
primary_key: true
- name: geographic_area_type
sql: geographic_area_type
type: string
- name: shape_clipped_json
sql: shape_clipped_json
type: string
```
When I use the dimension for grouping I get a query like this:
```sql
SELECT
geographic_area_shapes.shape_clipped_json,
SUM(some_col) AS ...
FROM ch_geography_silver.geographic_area_shapes
GROUP BY geographic_area_shapes.shape_clipped_json
```
What I would like to see is a way to force cube to generate SQL that looks like this instead:
```sql
SELECT
ANY(geographic_area_shapes.shape_clipped_json) AS shape_clipped_json,
SUM(some_col) AS ...
FROM ch_geography_silver.geographic_area_shapes
GROUP BY geographic_area_shapes.geographic_area_key # Using a stand-in for the grouping
```
This could be achieved with something like this:
```yaml
cubes:
- name: geographic_area_shapes
sql_table: '`ch_geography_silver`.`geographic_area_shapes`'
dimensions:
- name: geographic_area_key
sql: geographic_area_key
type: string
meta:
cube:
primary_key: true
- name: geographic_area_type
sql: geographic_area_type
type: string
- name: shape_clipped_json
sql: shape_clipped_json
type: string
group_replacement:
sql: {CUBE}.geographic_area_key
```
The bottom query is far more efficient in many database technologies when the shape_clipped_json column contains a large number of bytes. This is particularly true for columnar databases, where loading the shape column can be deferred until later on. In my narrow testing I was able to reduce query memory usage from 6 Gib to 3.9Gib, just by using a stand-in for the group by. The query also ran about 25% quicker.
**Describe alternatives you've considered**
I cannot think of a way to force the behaviour I am describing inside Cube
**Additional context**
This feature is useful in many contexts, where one column contains a lot of data, and another column maps 1:1 cleanly with it.
Contributor guide
Research direction
No source files or tests are identified in the issue. Start by tracing how dimension SQL is emitted in GROUP BY and ORDER BY clauses, then determine where a YAML group_replacement.sql setting would be parsed and applied. Done means the generated query groups by the replacement expression while retaining the dimension expression in SELECT, with regression coverage for the shown shape_clipped_json case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100