cockroachdb / cockroachdb/cockroach

optbuilder: DISTINCT ON doesn't work with expressions

Open
#107,516 4 comments 0 reactions 0 assignees View on GitHub
A-tools-hasura C-bug O-community T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

If one is writing a SQL query with a `DISTINCT ON`, where you are distincting over an case expression, in the following general form:

```sql
SELECT DISTINCT ON ()
FROM
ORDER BY
```
(the case expression in the `DISTINCT ON` and `ORDER BY` are identical)

then you will receive an error:

> SELECT DISTINCT ON expressions must match initial ORDER BY expressions

**To Reproduce**

Create the following data in CockroachDB:

```sql
CREATE TABLE "Characters"
(
"Id" integer NOT NULL,
"FirstName" text NOT NULL,
"Surname" text NOT NULL,
CONSTRAINT "Characters_pkey" PRIMARY KEY ("Id")
);

INSERT INTO "Characters" ("Id", "FirstName", "Surname") VALUES (1, 'Luke', 'Skywalker');
INSERT INTO "Characters" ("Id", "FirstName", "Surname") VALUES (2, 'Han', 'Solo');
INSERT INTO "Characters" ("Id", "FirstName", "Surname") VALUES (3, 'Leia', 'Organa');
INSERT INTO "Characters" ("Id", "FirstName", "Surname") VALUES (4, 'Obi-Wan', 'Kenobi');
```

then run the following query:

```sql
SELECT DISTINCT ON (
CASE
WHEN ("public"."Characters"."Id" = 1) THEN "Surname"
ELSE NULL
END
) *
FROM "public"."Characters"
ORDER BY
CASE
WHEN ("public"."Characters"."Id" = 1) THEN "Surname"
ELSE NULL
END DESC,
"Id" DESC
```

then you will receive an error:

> SELECT DISTINCT ON expressions must match initial ORDER BY expressions

If you extract the expression out to a column and then use it in `DISTINCT ON` and `ORDER BY`, it doesn't work either:

```sql
SELECT DISTINCT ON ("RedactedSurname")
*,
CASE
WHEN ("public"."Characters"."Id" = 1) THEN "Surname"
ELSE NULL
END AS "RedactedSurname"
FROM "public"."Characters"
ORDER BY
"RedactedSurname" DESC NULLS FIRST,
"Id" DESC NULLS FIRST
```

_However_, if you remove the `NULLS FIRST`, then the above query form works. Removing `NULLS FIRST` from the inlined case expression version doesn't work though.

**Expected behavior**
If you run the same query in Postgres, the query works and the expected result is returned.

![image](https://github.com/cockroachdb/cockroach/assets/1214352/0834a5e5-3d8b-4511-ac01-5a419af2db6a)

**Environment:**
- CockroachDB version: 23.1.6 (also a problem on v22.2.x too)
- Server OS: Linux
- Client app: `cockroach sql`

**Additional context**
This blocks us from using the same SQL generation logic we use for Postgres databases on Cockroach.

The above SQL examples are minimal reproductions of the problem. A more typical example of SQL we generate would be:

```sql
SELECT coalesce(
json_agg(
"root"
ORDER BY "root.pg.Surname" DESC NULLS FIRST,
"root.pg.Id" DESC NULLS FIRST
),
'[]'
) AS "root"
FROM (
SELECT DISTINCT ON ("root.pg.Surname") row_to_json(
(
SELECT "_e"
FROM (
SELECT "_root.base"."Id" AS "Id",
"_root.base"."FirstName" AS "FirstName",
CASE
WHEN ("_root.base"."Id") = ANY(
(ARRAY [('1')::bigint, ('2')::bigint])::bigint array
) THEN "_root.base"."Surname"
ELSE NULL
END AS "Surname"
) AS "_e"
)
) AS "root",
"_root.base"."Id" AS "root.pg.Id",
CASE
WHEN ("_root.base"."Id") = ANY(
(ARRAY [('1')::bigint, ('2')::bigint])::bigint array
) THEN "_root.base"."Surname"
ELSE NULL
END AS "root.pg.Surname"
FROM (
SELECT DISTINCT ON (
CASE
WHEN ("public"."Characters"."Id") = ANY(
(ARRAY [('1')::bigint, ('2')::bigint])::bigint array
) THEN "Surname"
ELSE NULL
END
) *
FROM "public"."Characters"
WHERE (
(
("public"."Characters"."Id") = ANY(
(ARRAY [('1')::bigint, ('2')::bigint])::bigint array
)
)
OR (
("public"."Characters"."Id") = ANY(
(
ARRAY [('2')::bigint, ('3')::bigint, ('4')::bigint, ('5')::bigint]
)::bigint array
)
)
)
ORDER BY CASE
WHEN ("public"."Characters"."Id") = ANY(
(ARRAY [('1')::bigint, ('2')::bigint])::bigint array
) THEN "Surname"
ELSE NULL
END DESC NULLS FIRST,
"Id" DESC NULLS FIRST
) AS "_root.base"
ORDER BY "root.pg.Surname" DESC NULLS FIRST,
"root.pg.Id" DESC NULLS FIRST
) AS "_root"
```

Jira issue: CRDB-30088

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the DISTINCT ON queries in cockroach sql, including the CASE expression and NULLS FIRST variants, and compare their behavior with PostgreSQL. Trace the optbuilder handling of DISTINCT ON and ORDER BY expressions. Done means the reported queries succeed and return the expected result without the expression-matching error.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.