MongoDB join perf optimization
- Dominant language
- Clojure
- Stars
- 49.3k
- Forks
- 6.8k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 653
Description
### Describe the bug
The way Metabase creates aggregation queries around filters + joins could be optimized quite a bit.
In particular, it's adding the $lookup + $unwind before the $match, even if the $match only applies to the initial collection.
This means that if the collections you're joining have millions of documents, but your match filters down to just 100 documents, Metabase asks MongoDB to fetch and join millions of documents before filtering down. Instead it could just fetch the 100 that actually matter, and then join.
### To Reproduce
1. Create a new question
2. Join with one or more other collections
3. Add a filter on the initial collection
4. Aggregation query will be:
```
$lookup + $unwind: ... (join)
$match: ... (filter)
```
### Expected behavior
Aggregation query should be:
```
$match: ... (filter)
$lookup + $unwind: ... (join)
```
### Logs
_No response_
### Information about your Metabase installation
```json
{
"browser-info": {
"language": "en-US",
"platform": "MacIntel",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.23+9-LTS",
"java.vendor": "Red Hat, Inc.",
"java.vendor.url": "https://www.redhat.com/",
"java.version": "11.0.23",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.23+9-LTS",
"os.name": "Linux",
"os.version": "4.14.348-265.565.amzn2.x86_64",
"user.language": "en",
"user.timezone": "UTC"
},
"metabase-info": {
"databases": [
"h2",
"mongo"
],
"hosting-env": "unknown",
"application-database": "h2",
"application-database-details": {
"database": {
"name": "H2",
"version": "2.1.214 (2022-06-13)"
},
"jdbc-driver": {
"name": "H2 JDBC Driver",
"version": "2.1.214 (2022-06-13)"
}
},
"run-mode": "prod",
"plan-alias": "",
"version": {
"date": "2024-07-30",
"tag": "v0.50.18",
"hash": "c323ffc"
},
"settings": {
"report-timezone": null
}
}
}
```
### Severity
annoying, but workaround available (rewrite queries manually)
### Additional context
Obviously if you are filtering on one of the joined collections the logic is a bit more complicated, but there's still significant optimization to be had... Let's say you join A → B and then B → C and filter on B... In that case it should:
```
$lookup + $unwind (join A to B)
$match (filter on B)
$lookup + $unwind (join B to C)
```
Instead it currently does:
```
$lookup + $unwind (join A to B)
$lookup + $unwind (join B to C)
$match (filter on B)
```
Contributor guide
Research direction
Reproduce the issue by creating a MongoDB question with a join and a filter on the initial collection, then inspect the generated aggregation pipeline. Trace the MongoDB query-generation entry point that produces the $lookup, $unwind, and $match stages; done means filters are applied before joins when safe, while filters on joined collections preserve the required join order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure, mongodb
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100