Custom question: Allow Filter before Join
Nobody has claimed this yet.
- Dominant language
- Clojure
- Stars
- 49.3k
- Forks
- 6.8k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 653
Description
## Problem
I have a questions in which I join a quite large dataset (~100M records) with a lookup table. However, I am only interested in a subset of the base database (say 1k records).
The current query takes quite some time (> 1 minute), and while I am also improving other parameters (db hardware, cache, indexes, materialized views, etc.) I want to streamline the join operation.
For that, **I would like to filter records from 100M to 1k, *and only then* join on 1k** — rather than join on 100M and then filter to 1k.
## Expected solution
**In the current GUI (v0.34.1), you can only filter the result of all the join operations** (i.e. join on 100M, then filter to 1k).
What I would like is to be able to:
1. Import a base data set (100M)
2. Apply a filter on this data set (100M → 1k)
3. Join the filtered base data set with another (e.g. lookup) table (1k)
4. [… continue with the rest of my question]
## Workarounds
Well, I can either:
* Filter after all the joins have been done (current natural solution): but this creates a resource heavy query, what I want to avoid;
* Import the base data set, apply the filter, create an aggregate that will actually return all the records, then join with the lookup table: this is just bulky, clunky, and feels wrong;
* Use a SQL query: but my point is to streamline the GUI query, without having user rely on SQL.
## Importance
I believe it's rather be a nice to have, as it's not blocking my workflow — just make things slow and my db suffer.
:arrow_down: Please click the :+1: reaction instead of leaving a `+1` or `update?` comment
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the GUI query flow for filters and joins, then determine where filter placement is represented before query generation. Done means users can apply a base-table filter before joining, the generated query avoids joining excluded rows, and the existing join workflow still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure
- Domain
- database, frontend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100