[Bug](nereids) DATEDIFF and all *_diff functions silently produce off-by-one results on varchar columns / subquery slots when session time_zone != UTC
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
## Version
apache/doris master at `f499c78c67c` (also affects `branch-4.0`, `branch-4.1`). Related previous fix `43800f3bf51` (PR #64127) is included, but is **insufficient** for the case described below.
## What's Wrong?
`DATEDIFF` (and every other `*_diff` date function) silently returns a wrong result whenever:
1. its argument is a **string-typed** value that is not a bare literal — e.g. a `varchar` column, a subquery projection slot, or a UNION-ALL output slot, **and**
2. the session `time_zone` is not `Etc/UTC`.
Under those two conditions Nereids binds the call to the `TIMESTAMPTZ` overload, which routes the value through `varchar → CAST(timestamptz(6))` and evaluates `daynr()` on the internal UTC-shifted representation, producing an off-by-one (or off-by-N) result.
The reason this has not been noticed in CI: master's default session `time_zone` is `Etc/UTC`, under which the varchar→timestamptz cast is a no-op shift and the wrong overload silently produces the right answer. **Any non-UTC deployment reproduces the bug immediately.**
### Reproducer against apache/master at `f499c78c67c`
```sql
-- master session default is Etc/UTC → looks fine
SELECT DATEDIFF(a, b)
FROM (SELECT '2025-08-01 00:00:00' AS a, '2025-05-30 11:40:09' AS b) t;
-- returns 63 ✓
-- flip to any non-UTC session tz (e.g. any China / East Asia deployment)
SET time_zone='+08:00';
SELECT DATEDIFF(a, b)
FROM (SELECT '2025-08-01 00:00:00' AS a, '2025-05-30 11:40:09' AS b) t;
-- returns 62 ✗ (off-by-one, WRONG)
-- Same bug on a real varchar column
CREATE TABLE td (a varchar(30), b varchar(30)) PROPERTIES('replication_num'='1');
INSERT INTO td VALUES ('2025-08-01 00:00:00', '2025-05-30 11:40:09');
SET time_zone='+08:00';
SELECT DATEDIFF(a, b) FROM td;
-- returns 62 ✗
-- Same bug in the shape that first surfaced this
SET time_zone='+08:00';
SELECT DATEDIFF(a, b)
FROM (
SELECT '2025-08-01 00:00:00' AS a, '2025-05-30 11:40:09' AS b
UNION ALL SELECT '2025-08-02 00:00:00', '2024-09-30 19:40:02'
) t;
-- returns 62 / 305 ✗ (expected 63 / 306)
```
`EXPLAIN VERBOSE` on any of the failing queries shows:
```
datediff(cast(a as TIMESTAMPTZ(6)), cast(b as TIMESTAMPTZ(6)))
```
## What You Expected?
`DATEDIFF(a, b)` should return the wall-clock day difference of the two input strings, **independent of session `time_zone`**. Expected values above: `63` and `63 / 306`.
## How to Reproduce?
1. Deploy any FE at master `f499c78c67c` (or `branch-4.0` / `branch-4.1`).
2. Connect via MySQL client.
3. Run the SQL statements above.
Reproduces deterministically on every non-UTC `time_zone`, tested on `Asia/Shanghai`, `+08:00`.
## Root Cause
Two independent facts combine into the bug:
### 1. Signature-order tie-break routes varchar to `TIMESTAMPTZ`
`fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/DateDiff.java`:
```java
public static final List SIGNATURES = ImmutableList.of(
FunctionSignature.ret(IntegerType.INSTANCE)
.args(TimeStampTzType.WILDCARD, TimeStampTzType.WILDCARD), // <- listed FIRST
FunctionSignature.ret(IntegerType.INSTANCE)
.args(DateTimeV2Type.WILDCARD, DateTimeV2Type.WILDCARD),
FunctionSignature.ret(IntegerType.INSTANCE)
.args(DateV2Type.INSTANCE, DateV2Type.INSTANCE));
```
`SearchSignature.doMatchTypes` only fires the `timeZoneCoersionScore--` penalty on `TIMESTAMPTZ` when the argument is a **literal it can inspect** — via `ExpressionUtils.getLiteralAfterUnwrapNullable(argument)` added by #64127. For a `SlotReference` typed `varchar` (which is what every subquery projection slot, UNION-ALL output slot, and real varchar column becomes at binding time), that helper returns `Optional.empty()`, so no scoring penalty fires. All three signatures then tie on `nonStrictMatchedWithoutStringLiteralCoercion` and the tie-break in `SearchSignature.result()` (lines 115–142) picks whichever candidate is seen first — **TIMESTAMPTZ**.
**#64127 only fixed the bare-literal / `Nullable(Literal)` subcase.** The much broader slot / column subcase is still latent.
### 2. `varchar → timestamptz(6)` cast on non-UTC session shifts the value
Per `be/src/exprs/function/cast/cast_to_timestamptz_impl.hpp`:
> When timezone info is absent, `TIMESTAMP_TZ` treats input as local time and converts to UTC
And `TimestampTzValue::daynr()` returns `_utc_dt.daynr()` — the daynr of the shifted UTC value. So `'2025-08-01 00:00:00'` under session `+08:00` becomes UTC `2025-07-31 16:00:00`, and its `daynr()` is one less than the wall-clock date.
The signature is picked at bind time and the shift happens at cast time; both paths agree even when `debug_skip_fold_constant=true`, so the bug is not a folding artifact.
## Scope
This same signature ordering exists in **12 date functions**, so all of them are affected identically:
```
DateDiff, DaysDiff, HoursDiff, MicroSecondsDiff, MilliSecondsDiff,
MinutesDiff, MonthsDiff, QuartersDiff, SecondsDiff, TimeDiff,
WeeksDiff, YearsDiff
```
## Suggested Fix
Reorder the `SIGNATURES` list in the 12 `*Diff.java` files to put `TIMESTAMPTZ` **last**:
```java
public static final List SIGNATURES = ImmutableList.of(
FunctionSignature.ret(IntegerType.INSTANCE)
.args(DateTimeV2Type.WILDCARD, DateTimeV2Type.WILDCARD),
FunctionSignature.ret(IntegerType.INSTANCE)
.args(DateV2Type.INSTANCE, DateV2Type.INSTANCE),
FunctionSignature.ret(IntegerType.INSTANCE)
.args(TimeStampTzType.WILDCARD, TimeStampTzType.WILDCARD)); // <- moved LAST
```
### Why this is correct (traced through `SearchSignature`)
- `DATEDIFF(datetimev2_col, datetimev2_col)` — `DateTimeV2` matches identically (`nonStrictMatched=0`); still wins. ✓
- `DATEDIFF(timestamptz_col, timestamptz_col)` — `TIMESTAMPTZ` matches identically; still wins even when listed last. ✓
- `DATEDIFF(date_col, date_col)` — `DateV2` matches identically; still wins. ✓
- `DATEDIFF('str+08:00', 'str+08:00')` — literal-with-TZ still routes to `TIMESTAMPTZ` because `timeZoneCoersionScore=+1` beats `0` in `doMatchTypes` regardless of list order. ✓
- `DATEDIFF('2025-08-01 00:00:00', '2025-05-30 11:40:09')` — literal-without-TZ: `TIMESTAMPTZ` gets `-1`, `DateTimeV2` gets `0`, `DateTimeV2` wins. ✓ (matches #64127's intent)
- `DATEDIFF(varchar_col, varchar_col)` — **the failing case**: all three sigs tie; first candidate wins → now `DateTimeV2` → wall-clock semantics → correct. ✓
Verified locally against apache/master at `f499c78c67c` with the reorder applied to just `DateDiff.java`:
```sql
SET time_zone='+08:00'; SELECT DATEDIFF(a,b) FROM td; -- 63 ✓
SET time_zone='Asia/Shanghai'; SELECT DATEDIFF(a,b) FROM td; -- 63 ✓
SET time_zone='America/Los_Angeles'; SELECT DATEDIFF(a,b) FROM td; -- 63 ✓
SET time_zone='+14:00'; SELECT DATEDIFF(a,b) FROM td; -- 63 ✓
SET time_zone='-12:00'; SELECT DATEDIFF(a,b) FROM td; -- 63 ✓
```
Happy to send a PR with the 12-file reorder + regression tests covering (a) varchar column, (b) subquery slot, (c) UNION-ALL slot, and (d) all shapes under both UTC and non-UTC session tz.
## Anything Else?
- Prior related fix: #64127 (fixes bare-literal and `Nullable(Literal)` cases only).
- Related infrastructure: the `matchType` scoring introduced in #57586, and the TIMESTAMPTZ overloads added in #59206.
- Not caused by folding — reproduces identically with `SET debug_skip_fold_constant=true`.
## Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
## Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/DateDiff.java, then inspect the other 11 *Diff.java files and SearchSignature.doMatchTypes/result(). Reproduce the varchar-column and subquery cases with a non-UTC session time_zone, reorder the signatures as described, and add regression coverage for the listed input shapes and time zones. Done means the expected wall-clock differences are returned consistently without breaking typed date or timestamp arguments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100