metabase / metabase/metabase

Custom question: Allow Filter before Join

Open
#11,869 8 comments 20 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Querying/Notebook Type:New Feature
Dominant language
Clojure
Stars
49.3k
Forks
6.8k
Avg merge
1d 13h
Merged PRs (30d)
653

Description

## Problem
I have a questions in which I join a quite large dataset (~100M records) with a lookup table. However, I am only interested in a subset of the base database (say 1k records).

The current query takes quite some time (> 1 minute), and while I am also improving other parameters (db hardware, cache, indexes, materialized views, etc.) I want to streamline the join operation.
For that, **I would like to filter records from 100M to 1k, *and only then* join on 1k** — rather than join on 100M and then filter to 1k.

## Expected solution
**In the current GUI (v0.34.1), you can only filter the result of all the join operations** (i.e. join on 100M, then filter to 1k).

What I would like is to be able to:
1. Import a base data set (100M)
2. Apply a filter on this data set (100M → 1k)
3. Join the filtered base data set with another (e.g. lookup) table (1k)
4. [… continue with the rest of my question]

## Workarounds
Well, I can either:
* Filter after all the joins have been done (current natural solution): but this creates a resource heavy query, what I want to avoid;
* Import the base data set, apply the filter, create an aggregate that will actually return all the records, then join with the lookup table: this is just bulky, clunky, and feels wrong;
* Use a SQL query: but my point is to streamline the GUI query, without having user rely on SQL.

## Importance
I believe it's rather be a nice to have, as it's not blocking my workflow — just make things slow and my db suffer.

:arrow_down: Please click the :+1: reaction instead of leaving a `+1` or `update?` comment

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the GUI query flow for filters and joins, then determine where filter placement is represented before query generation. Done means users can apply a base-table filter before joining, the generated query avoids joining excluded rows, and the existing join workflow still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
database, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.