cockroachdb / cockroachdb/cockroach
Slow select query using tuple filter clause
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
I have a select query that selects from a large table (100s million rows), joining on 2 columns, and filtering on 2 other columns. When I used separate IN filter clauses, ie `column1 IN (...) AND column2 IN (...)`, the query runs quickly as expected (roughly 0.05s for 1000s of rows). However, when I query the same rows using a single filter clause, ie `(column1, column2) IN (...)` the query becomes over 200x slower. Our table has a multi index that includes all 4 columns.
For example, if the fast query follows the pattern: `... column1 IN (1, 2) AND column2 IN (A, B) ...` then the slow query looks like `... WHERE (column1, column2) IN ((1, A), (1, B), (2, A), (2, B)) ...`.
When benchmarking this, I am querying for the same exact rows (in the magnitude of 1000s of rows) to get an apples to apples benchmark. So if `x` is a list of column 1 values and `y` is a list of column 2 values, then both queries are searching for the cross product of `x` and `y`.
In practice, the use for the "slower" query is if we do not want a full cross product, but rather a more specific subset. The workaround I am currently using is to always use the fast type of query, and then filtering out unneeded values in python afterward. This seems like extra work that should not be needed though.
**To Reproduce**
1. Set up a cockroach db instance.
2. Follow the code below to repro the issue. I used [`sqlalchemy`](https://www.sqlalchemy.org/) and [`sqlalchemy_cockroachdb`](https://pypi.org/project/sqlalchemy-cockroachdb/), along with an ipython notebook to create the example repro, but once you create the schema and insert in some dummy data it is reproducible directly using sql code as well.
Import dependencies
```python
import collections
import itertools
import uuid
import sqlalchemy
import sqlalchemy.orm
import sqlalchemy_cockroachdb
```
Create db connection using the db instance url
```python
engine = sqlalchemy.create_engine(url, isolation_level="SERIALIZABLE")
sessionmaker = sqlalchemy.orm.sessionmaker(engine, expire_on_commit=False)
```
Create the table
```python
sqlalchemy_cockroachdb.run_transaction(
sessionmaker,
lambda session: session.execute(
"""
CREATE TABLE repro (
item_id UUID NOT NULL,
group_id UUID NOT NULL,
item_name STRING,
subgroup INT NOT NULL,
PRIMARY KEY (item_id, group_id, item_name, subgroup)
)
"""
)
)
```
Generate some dummy data. I am generating more data than needed so that the total size of the table makes the benchmark timing more apparent.
```python
def generate_data():
# Track the item ids on each group for querying later
item_ids_by_group = collections.defaultdict(list)
# We have a relatively small number of groups
for _ in range(10):
group_id = uuid.uuid4()
# Each group has on the order of 100s of subgroups
for subgroup in range(100):
rows = []
# Each subgroup contains some items
for _ in range(10):
item = uuid.uuid4()
item_ids_by_group[group_id].append(item)
# Each item is subdivided with item names
for ii in range(100):
rows.append((item, f"item_{ii}"))
# Insert values in reasonable sized batches
sql_values = ",".join([f"('{item_id}'::uuid, '{group_id}'::uuid, '{item_name}', {subgroup})" for item_id, item_name in rows])
sqlalchemy_cockroachdb.run_transaction(
sessionmaker,
lambda session: session.execute(
f"INSERT INTO repro (item_id, group_id, item_name, subgroup) VALUES {sql_values}"
)
)
generate_data()
```
Grab the subset of data that we will be querying for.
```python
# Search within specific groups
groups = [
row.group_id for row in
sqlalchemy_cockroachdb.run_transaction(
sessionmaker,
lambda session: session.execute(
"SELECT DISTINCT group_id FROM repro LIMIT 2"
).all(),
)
]
# Search for specific item ids
item_ids = [
row.item_id for row in
sqlalchemy_cockroachdb.run_transaction(
sessionmaker,
lambda session: session.execute(
f"SELECT DISTINCT item_id FROM repro WHERE group_id IN ('{groups[0]}'::UUID, '{groups[1]}'::UUID) LIMIT 300"
).all(),
)
]
```
This is the query that repros the issue. As you'll see this query is surprisingly slow.
```python
# The query basically uses a join to filter for all subgroups belonging to a couple of specific group ids.
# And then uses a where clause to filter for the cross product specific item ids x item names.
# NOTE: This takes a while to run.
# NOTE: This issue only seems to pop up when the total number of rows is above a certain threshold (not sure exactly how many, but it's in the 1000s range).
# Get the cross product item ids x item names
cross_product_rows = list(itertools.product(item_ids, [f"item_{ii}" for ii in range(100)]))
sql_filter = ",".join(f"('{item_id}'::UUID, '{item_name}')" for item_id, item_name in cross_product_rows)
sqlalchemy_cockroachdb.run_transaction(
sessionmaker,
lambda session: session.execute(
f"""
EXPLAIN ANALYZE
WITH groups (group_id, subgroup) AS (
VALUES ('{groups[0]}'::UUID, 99),('{groups[1]}'::UUID, 99)
)
SELECT DISTINCT ON (item_id, item_name)
item_id,
repro.group_id,
item_name,
repro.subgroup
from repro
JOIN groups
ON repro.group_id = groups.group_id
AND repro.subgroup <= groups.subgroup
WHERE (item_id, item_name) in ({sql_filter})
ORDER BY
item_id,
item_name,
repro.subgroup DESC
"""
).all(),
)
```
This is the first few lines of the explain analyze for this query. The execution time is very long.
```
planning time: 352ms
execution time: 10.6s
distribution: local
vectorized: true
rows read from KV: 30,000 (2.1 MiB, 1 gRPC calls)
cumulative time spent in KV: 51ms
maximum memory usage: 9.2 MiB
network usage: 0 B (0 messages)
```
Compare that to this query which uses separate filter clauses but queries for the exact same rows:
```python
item_ids_sql = ",".join(f"'{item_id}'::UUID" for item_id in item_ids)
item_names_sql = ",".join([f"'item_{ii}'" for ii in range(100)])
res = sqlalchemy_cockroachdb.run_transaction(
sessionmaker,
lambda session: session.execute(
f"""
EXPLAIN ANALYZE
WITH groups (group_id, subgroup) AS (
VALUES ('{groups[0]}'::UUID, 99),('{groups[1]}'::UUID, 99)
)
SELECT DISTINCT ON (item_id, item_name)
item_id,
repro.group_id,
item_name,
repro.subgroup
from repro
JOIN groups
ON repro.group_id = groups.group_id
AND repro.subgroup <= groups.subgroup
WHERE item_id IN ({item_ids_sql})
AND item_name IN ({item_names_sql})
ORDER BY
item_id,
item_name,
repro.subgroup DESC
"""
).all(),
)
```
Here is the first few lines of the explain analyze output. It is the same resulting rows, but 200x faster.
```
planning time: 933µs
execution time: 64ms
distribution: full
vectorized: true
rows read from KV: 30,000 (2.1 MiB, 1 gRPC calls)
cumulative time spent in KV: 51ms
maximum memory usage: 7.1 MiB
network usage: 0 B (0 messages)
```
**Environment:**
- CockroachDB version 22.2.5
Jira issue: CRDB-27418
Contributor guide
Assessment
This issue has not been assessed yet.