BLOB prefix-index semi-join returns wrong results (missed value / duplicated empty-string matches) when tables have row counts but no histograms
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
```sql
DROP DATABASE IF EXISTS tidb_repro_prefix_index;
CREATE DATABASE tidb_repro_prefix_index;
USE tidb_repro_prefix_index;
CREATE TABLE t0(c0 BLOB(54) NOT NULL);
INSERT INTO t0 VALUES (UNHEX('')),(UNHEX('')),(UNHEX('302E36313136373338343339313134343736')),(UNHEX('2D5E097B69')),(UNHEX('2D31353234393434393733')),(UNHEX('3147')),(UNHEX('')),(UNHEX('534A4959EC92A54F5075'));
CREATE TABLE t1(c0 BLOB(54) NOT NULL);
INSERT INTO t1 VALUES (UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('')),(UNHEX('2D31353234393434393733'));
CREATE INDEX i0 ON t0(c0(58));
-- The buggy IndexJoin+StreamAgg+IndexRangeScan plan is picked only when table-level row
-- counts are known while column histograms are absent (the state ANALYZE + DROP STATS
-- leaves behind). On a never-analyzed table (pure pseudo stats) the optimizer picks
-- HashJoin and the bug does not show.
ANALYZE TABLE t0, t1;
DROP STATS t0;
DROP STATS t1;
-- Base query (IN in the WHERE predicate):
SELECT t1.c0 FROM t1 WHERE t1.c0 IN (SELECT t0.c0 FROM t0 WHERE true);
-- 32 rows (32 empty strings) <-- WRONG (expected 33: the '-1524944973' row is missing)
-- Rewritten query (same IN relocated into the derived-table projection):
SELECT ref0 FROM (SELECT t1.c0 AS ref0, t1.c0 IN (SELECT t0.c0 FROM t0 WHERE true) AS ref1 FROM t1) AS s WHERE ref1;
-- 33 rows (32 empty strings + '-1524944973') <-- correct
```
Notes on the trigger conditions (each verified experimentally):
- **Prefix index on the BLOB column is required**: `DROP INDEX` or the `IGNORE_INDEX(t0, i0)` hint makes the base query return the correct 33 rows. Prefix length is irrelevant (`c0(58)`, `c0(54)`, `c0(11)` all reproduce).
- **Statistics state matters**: the wrong plan is chosen only in the row-counts-without-histograms state (`ANALYZE` then `DROP STATS`). With a never-analyzed table (pure pseudo stats) the optimizer picks HashJoin and the result is correct. An `INL_JOIN` hint forcing the IndexJoin shape does *not* reproduce on un-analyzed tables — the stats state affects more than plan choice.
- **The data itself is load-bearing**: replacing the eight `t0` rows above with a semantically equivalent hand-written set (3 empty strings, `-1524944973`, four filler uniques) does not reproduce; the exact byte values matter. `t1` only needs the 32 empty strings plus `-1524944973` (rows whose values match nothing were removed).
- **A second manifestation** observed during fuzzing with slightly different data: the empty-string rows are returned **twice** — `t1` holding 32 `''` rows, the base query returned 64. `EXPLAIN ANALYZE` showed the rewritten inner-side `StreamAgg` (the de-dup) emitting **2 rows for the single value `''`**, so each `''` probe row matched twice.
- Not a duplicate of #70546 (also IN semi-join, but already fixed on this build and a different symptom).
### 2. What did you expect to see? (Required)
Both queries return the same 33 rows (32 empty strings + `-1524944973`, which exists in `t0` and therefore satisfies the IN). `IN` is a deterministic predicate: evaluating it in the WHERE clause or materializing it into a derived-table projection must produce identical results.
### 3. What did you see instead? (Required)
The base query returns **32 rows**: the `-1524944973` row is missing. The rewritten query returns the correct 33 rows.
`EXPLAIN` of the base query:
```
Projection_12
└─IndexJoin_17 inner join, inner:StreamAgg_39, outer key:t1.c0, inner key:t0.c0, equal cond:eq(t1.c0, t0.c0)
├─TableReader_29(Build) data:TableFullScan_28 (t1)
└─StreamAgg_39(Probe) group by:t0.c0
└─IndexLookUp_57
├─IndexRangeScan_55(Build) table:t0, index:i0(c0) range: decided by [eq(t0.c0, t1.c0)]
└─TableRowIDScan_56(Probe)
```
The correct plan (chosen when the index is dropped/ignored) is `IndexHashJoin (left outer semi join)`. `IGNORE_INDEX(t0, i0)` alone fixes the base query, confirming the wrong result comes from the prefix-index `IndexRangeScan` path of the IN → semi/inner-join rewrite.
### 4. What is your TiDB version? (Required)
```
Release Version: v9.0.0-beta.2.pre-2148-g0419c93284
Edition: Community
Git Commit Hash: 0419c93284fb3cf1b08630ba3837b5f9df4c5c5d
Git Branch: HEAD
UTC Build Time: 2026-08-25 07:21:24
GoVersion: go1.25.12
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Kernel Type: Classic
```
Contributor guide
Research direction
Start by running the SQL reproducer after ANALYZE TABLE and DROP STATS, then compare EXPLAIN and EXPLAIN ANALYZE for the base and rewritten queries. Trace the IN-to-semi-join plan through IndexJoin, StreamAgg, and the prefix-index IndexRangeScan path. Done means both queries return the same 33 rows and the duplicate empty-string manifestation is absent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100