metabase / metabase/metabase

MongoDB join perf optimization

Open
#49,643 0 comments 0 reactions 0 assignees View on GitHub
.Performance .Project candidate .Team/QueryingPlatform Database/Mongo Type:New Feature
Dominant language
Clojure
Stars
49.3k
Forks
6.8k
Avg merge
1d 13h
Merged PRs (30d)
653

Description

### Describe the bug

The way Metabase creates aggregation queries around filters + joins could be optimized quite a bit.

In particular, it's adding the $lookup + $unwind before the $match, even if the $match only applies to the initial collection.

This means that if the collections you're joining have millions of documents, but your match filters down to just 100 documents, Metabase asks MongoDB to fetch and join millions of documents before filtering down. Instead it could just fetch the 100 that actually matter, and then join.

### To Reproduce

1. Create a new question
2. Join with one or more other collections
3. Add a filter on the initial collection
4. Aggregation query will be:

```
$lookup + $unwind: ... (join)
$match: ... (filter)
```

### Expected behavior

Aggregation query should be:

```
$match: ... (filter)
$lookup + $unwind: ... (join)
```

### Logs

_No response_

### Information about your Metabase installation

```json
{
"browser-info": {
"language": "en-US",
"platform": "MacIntel",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.23+9-LTS",
"java.vendor": "Red Hat, Inc.",
"java.vendor.url": "https://www.redhat.com/",
"java.version": "11.0.23",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.23+9-LTS",
"os.name": "Linux",
"os.version": "4.14.348-265.565.amzn2.x86_64",
"user.language": "en",
"user.timezone": "UTC"
},
"metabase-info": {
"databases": [
"h2",
"mongo"
],
"hosting-env": "unknown",
"application-database": "h2",
"application-database-details": {
"database": {
"name": "H2",
"version": "2.1.214 (2022-06-13)"
},
"jdbc-driver": {
"name": "H2 JDBC Driver",
"version": "2.1.214 (2022-06-13)"
}
},
"run-mode": "prod",
"plan-alias": "",
"version": {
"date": "2024-07-30",
"tag": "v0.50.18",
"hash": "c323ffc"
},
"settings": {
"report-timezone": null
}
}
}
```

### Severity

annoying, but workaround available (rewrite queries manually)

### Additional context

Obviously if you are filtering on one of the joined collections the logic is a bit more complicated, but there's still significant optimization to be had... Let's say you join A → B and then B → C and filter on B... In that case it should:

```
$lookup + $unwind (join A to B)
$match (filter on B)
$lookup + $unwind (join B to C)
```

Instead it currently does:

```
$lookup + $unwind (join A to B)
$lookup + $unwind (join B to C)
$match (filter on B)
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue by creating a MongoDB question with a join and a filter on the initial collection, then inspect the generated aggregation pipeline. Trace the MongoDB query-generation entry point that produces the $lookup, $unwind, and $match stages; done means filters are applied before joins when safe, while filters on joined collections preserve the required join order.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure, mongodb
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.