sql: ordinals nested in a parenthesized GROUP BY item are not resolved as select-list positions

## Describe the problem

CockroachDB resolves select-list ordinals against the *whole* `GROUP BY` item
before flattening parenthesized lists. PostgreSQL flattens first, then resolves
each resulting element. So an integer nested inside a parenthesized `GROUP BY`
list — `GROUP BY (1, x)` — stays a literal constant in CockroachDB, where
PostgreSQL reads it as select-list position 1.

Bare ordinals (`GROUP BY 1`) and singly/doubly parenthesized ones
(`GROUP BY (1)`, `GROUP BY ((1))`) agree, because `tree.StripParens` unwraps
those to a bare integer before ordinal resolution. The divergence appears only
once the item is a multi-element tuple, which `StripParens` leaves alone.

This diverges in both directions — CockroachDB accepts queries PostgreSQL
rejects, and rejects queries PostgreSQL accepts.

## To Reproduce

```sql
CREATE TABLE ab (k INT, v INT);
INSERT INTO ab VALUES (1, 10);
```

**Direction 1 — CockroachDB accepts what PostgreSQL rejects.** The nested `1`
should resolve to select item 1, which is an aggregate and therefore illegal as
a grouping key:

```sql
SELECT min(b.k), a.k, a.v
FROM (VALUES (1,10)) AS a(k,v),
(VALUES (2)) AS b(k)
GROUP BY (1, (a.*));
```

* PostgreSQL: `ERROR 42803: aggregate functions are not allowed in GROUP BY`
* CockroachDB: **accepted**, returns `2 | 1 | 10`

**Direction 2 — CockroachDB rejects what PostgreSQL accepts.** Ordinary
positional grouping inside parens:

| Query | PostgreSQL 19devel | CockroachDB |
| --- | --- | --- |
| `SELECT k FROM ab GROUP BY (1, v);` | 1 row (`1`) | **`ERROR 42803: column "k" must appear in the GROUP BY clause…`** |
| `SELECT k, v FROM ab GROUP BY (2, 1);` | 1 row (`1 \| 10`) | **`ERROR 42803: column "k" must appear in the GROUP BY clause…`** |

**Direction 3 — same outcome, wrong reason.** Both reject, but CockroachDB
never sees the ordinal, so it reports the wrong problem:

```sql
SELECT min(b.k), a.k, a.v
FROM (VALUES (1,10)) AS a(k,v), (VALUES (2)) AS b(k)
GROUP BY (1, 2);
```

* PostgreSQL: `ERROR 42803: aggregate functions are not allowed in GROUP BY`
* CockroachDB: `ERROR 42803: column "k" must appear in the GROUP BY clause…`

**Cases that already agree**, for contrast — all resolve the ordinal correctly:

| Query | Both |
| --- | --- |
| `SELECT min(b.k), a.k FROM … GROUP BY 1;` | `ERROR: aggregate functions are not allowed in GROUP BY` |
| `SELECT k FROM ab GROUP BY (1);` | 1 row (`1`) |
| `SELECT k FROM ab GROUP BY ((1));` | 1 row (`1`) |

## Expected behavior

Parenthesized `GROUP BY` lists should be flattened *before* select-list ordinal
and alias resolution, so that nested integer constants are resolved as ordinals
exactly as bare ones are — matching PostgreSQL. `GROUP BY (2, 1)` should mean
`GROUP BY 2, 1`.

## Additional context

**Why it happens.** In `buildGrouping`
(`pkg/sql/opt/optbuilder/groupby.go`) the pipeline order is:

1. `tree.StripParens(groupBy)` — unwraps `((a))` to `a`, which is why the
single-element paren cases work.
2. `colIndex(len(selects), groupBy, "GROUP BY")` — ordinal resolution, applied
to the top-level item. For a multi-element tuple this sees a `*tree.Tuple`,
not an integer, so it returns -1 and no ordinal resolution happens.
3. `b.expandStarAndResolveType(...)` then `flattenTuples(...)` — flattening,
which finally exposes the nested `1`, long after the only place that would
have interpreted it as an ordinal.

PostgreSQL's `transformGroupClause` runs in the opposite order: it unwraps
implicit `RowExpr`s first, then calls `findTargetListEntrySQL92` on each
resulting element. The comment block above `colIndex` in `groupby.go` is
actually pasted verbatim from that PostgreSQL function — including "2.
IntegerConstant — This means to use the n'th item in the existing target list" —
so the intent is to match; only the position in the pipeline differs.

**This is not a conformance defect.** The SQL standard sanctions neither side of
this behavior, so it cannot adjudicate. ISO/IEC 9075-2 subclause 7.9 admits only
column references as grouping items:

```
::=

|

::= [ ]
```

Positional references have never been standard in `GROUP BY` — not in SQL-92
either — and the standard has since dropped them from `ORDER BY` as well
(subclause 10.10 is now ` ::= `, with no
`` alternative). But by the same production, arbitrary scalar
*expressions* in `GROUP BY` are equally non-standard, and every major engine
ships them regardless.

So both `GROUP BY 1` and `GROUP BY a + b` are universal vendor extensions with
no conforming behavior to measure against. PostgreSQL compatibility is the only
meaningful yardstick here, which is what this report argues from. The comment
pasted into `groupby.go` from PostgreSQL concedes as much: "GROUP BY
column-number is not allowed by SQL92, but since the standard has no other
behavior defined for this syntax, we may as well accept this common extension."

**Secondary finding: `a.*` in `GROUP BY`.** The reproduction above relies on a
second, independent divergence. PostgreSQL treats `a.*` in `GROUP BY` as a
whole-row reference that does not make the individual columns available;
CockroachDB expands it into its components (step 3 above,
`expandStarAndResolveType`):

```sql
SELECT a.k, a.v FROM (VALUES (1,10)) AS a(k,v) GROUP BY a.*; -- PG: ERROR 42803; CRDB: 1 | 10
SELECT a.k, a.v FROM (VALUES (1,10)) AS a(k,v) GROUP BY (a.*); -- PG: ERROR 42803; CRDB: 1 | 10
```

This is arguably a separate bug and could be split out.

**Related.** #173264 and #173468 also stem from how `GROUP BY` items are routed
through generic scalar-expression handling in this same function, though the
fixes are independent.

## Environment

* CockroachDB v26.4.0-alpha (master, August 2026), single-node `cockroach demo`
* Compared against PostgreSQL 19devel

Jira issue: CRDB-66788

Start in pkg/sql/opt/optbuilder/groupby.go, especially buildGrouping and the order of StripParens, colIndex, expandStarAndResolveType, and flattenTuples. Reproduce the listed nested GROUP BY queries against CockroachDB and PostgreSQL; done means parenthesized lists resolve integer items as select-list positions with PostgreSQL-compatible results, while the separate a.* behavior remains distinguishable.

Written by the indexing model from the issue text.