matrixorigin / matrixorigin/matrixone
[Bug]: GROUP BY and distinct aggregates split CHAR trailing-space values that compare equal
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
`GROUP BY`, `SELECT DISTINCT`, and distinct aggregates can split fixed-width `CHAR` values that differ only in trailing spaces, even though MatrixOne's comparison and unique-key paths treat those values as equal.
For the four stored values `'a'`, `'a '`, `'a '`, and `'a '` in a `CHAR(4)` column:
- `c = 'a'` is true for all four rows;
- `WHERE c = 'a'` returns all four rows;
- a unique index rejects the second spelling as a duplicate;
- `COUNT(DISTINCT c)`, `APPROX_COUNT_DISTINCT(c)`, and `HLL_ADD_AGG(c)` each return cardinality `4` instead of `1`;
- `GROUP BY c` and `SELECT DISTINCT c` produce four keys instead of one.
This violates the SQL equivalence invariant inside MatrixOne: values that compare equal and conflict in a unique key must not become separate grouping/distinct keys.
## Environment
- MatrixOne: latest official `main`, commit `07fdd4ae0f80f287b93fc4b525fd296d5617abc7`
- Deployment: local standalone launch (isolated Log/TN/CN ports and data directory)
## Reproduction
```sql
drop database if exists char_distinct_group_repro;
create database char_distinct_group_repro;
use char_distinct_group_repro;
create table t(id int, c char(4));
insert into t values
(1, 'a'),
(2, 'a '),
(3, 'a '),
(4, 'a ');
select id, c = 'a' as equals_a from t order by id;
select count(*) from t where c = 'a';
select count(distinct c) from t;
select count(*) from (select distinct c from t) d;
select count(*) from (select c from t group by c) g;
create table u(c char(4), unique key uk(c));
insert into u values ('a');
insert into u values ('a ');
```
## Actual behavior
```text
equals_a: 1, 1, 1, 1
WHERE count: 4
COUNT DISTINCT: 4
SELECT DISTINCT groups: 4
GROUP BY groups: 4
second unique insert: Duplicate entry 'a ' for key 'uk'
```
The result was reproduced three times from freshly recreated tables on the same `main` build.
A width matrix further isolates two affected paths:
```text
type COUNT(DISTINCT c) GROUP BY groups rows matching c = 'a'
CHAR(4) 4 4 4
CHAR(8) 4 1 4
CHAR(16) 4 1 4
CHAR(32) 4 1 4
```
Thus the short-key grouping path is affected, while the distinct-aggregate path is affected independently of these tested widths.
For both `CHAR(4)` and `CHAR(32)`, `APPROX_COUNT_DISTINCT(c)` and `HLL_CARDINALITY(HLL_ADD_AGG(c))` also return `4`. A `VARCHAR(4)` control returns four across all grouping/distinct paths, as expected because its trailing spaces remain significant. `COUNT(DISTINCT c, 7)` returns four for both CHAR widths, so multi-argument exact DISTINCT is affected too.
## Expected behavior
All three distinct/grouping forms should produce one key, consistent with `=`, `WHERE`, and the unique index.
## Code-path analysis
The behavior depends on the internal hash-map representation rather than SQL semantics:
- `pkg/common/hashmap/strhashmap.go` calls `canonicalVarlenaHashValue`, which trims trailing ASCII spaces for `types.T_char`;
- `pkg/common/hashmap/inthashmap.go:fillVarlenaKey` copies raw varlena bytes directly and does not apply the same `CHAR` canonicalization; short grouping keys can therefore receive different group IDs;
- exact distinct-aggregate admission and retained-key logic in `pkg/sql/colexec/aggexec` compares/copies raw argument bytes, with special canonicalization only for floating signed zero; it has no PAD SPACE canonicalization for `CHAR`;
- HLL/approximate distinct aggregation hashes raw argument bytes and likewise lacks typed `CHAR` canonicalization.
This explains why a `StrHashMap`-only unit test protects longer SQL grouping keys but not the `IntHashMap` or distinct-aggregate paths.
## Regression context
#28023 recorded a related PAD SPACE grouping defect and was closed after a prior `main` build grouped the values together. The current official `main` has regressed the SQL grouping/distinct behavior through a different short-key hash path, so this report is new rather than reopening the closed issue.
## Suggested regression coverage
- `IntHashMap` unit coverage for `CHAR` values with zero through full-width trailing spaces, including nullable keys;
- SQL `GROUP BY`, `SELECT DISTINCT`, exact/approximate/HLL distinct aggregates for both short and long `CHAR` widths;
- DOP 1/DOP >1 and resident/spill aggregate paths;
- controls for `VARCHAR`, whose trailing spaces remain significant.
Contributor guide
Assessment
This issue has not been assessed yet.