apache / apache/doris

[Bug] Year-zero DATETIME survives the Parquet write but is rejected on read (MIN_DORIS_TIMESTAMP_MICROS stops at 0001-01-01)

Open
#67,447 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

master (`ddbaaab1388`)

### What's Wrong?

Doris writes year-zero `DATETIME` values to Parquet correctly, but cannot read its own file back — the values come back `NULL`.

`be/src/core/data_type_serde/parquet_timestamp.h` gates every Parquet timestamp read at year 1:

```cpp
inline constexpr int64_t MIN_DORIS_TIMESTAMP_MICROS = -62135596800000000LL; // 0001-01-01
inline constexpr int64_t MAX_DORIS_TIMESTAMP_MICROS = 253402300799999999LL; // 9999-12-31

inline Status validate_parquet_timestamp_micros(int64_t timestamp_micros) {
if (timestamp_micros < MIN_DORIS_TIMESTAMP_MICROS || timestamp_micros > MAX_DORIS_TIMESTAMP_MICROS) {
return Status::DataQualityError(
"Parquet timestamp is outside the Doris 0001-9999 range: micros={}", timestamp_micros);
}
return Status::OK();
}
```

But the DATETIME type itself starts at `0000-01-01`, whose micros value is `-62167219200000000` — below that bound. So the reader is stricter than the type it materialises into.

### What You Expected?

A `DATETIME` value that Doris accepts, stores and exports should read back from Doris's own Parquet file.

### How to Reproduce?

```sql
CREATE TABLE dt0 (id INT, ts DATETIME(6)) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1');

INSERT INTO dt0 VALUES (1,'0000-01-01 12:34:56'),(2,'0000-03-01 00:00:00'),
(3,'1969-12-31 23:59:59'),(4,'2024-01-01 12:00:00');

-- the type stores them fine
SELECT id, CAST(ts AS STRING) FROM dt0 ORDER BY id;
-- 1 0000-01-01 12:34:56.000000
-- 2 0000-03-01 00:00:00.000000
-- 3 1969-12-31 23:59:59.000000
-- 4 2024-01-01 12:00:00.000000

SELECT * FROM dt0 ORDER BY id INTO OUTFILE 'file:///tmp/dtz/p_' FORMAT AS PARQUET;
```

Read the file back with the `local()` TVF:

```
id ts_read
1 NULL <-- lost
2 NULL <-- lost
3 1969-12-31 23:59:59.000000
4 2024-01-01 12:00:00.000000
```

Note `0000-03-01` is also lost, so this is a plain range check, not a calendar edge case.

### Anything Else?

**The write side is correct**: Spark reads the same year-zero timestamps out of a Doris-written Iceberg table without trouble (verified with `apache/spark:4.0.0` against the Iceberg REST catalog — Spark returned `0000-01-01 12:34:56` for the row Doris returned `NULL` for). So only Doris's reader rejects them.

The constant name and the error text both say `0001-9999`, so the bound may be deliberate. If so, the asymmetry is still worth resolving in one direction or the other:

- widen the reader to `0000-01-01` (`-62167219200000000`) to match the DATETIME range, or
- reject year-zero timestamps on the write path too, so Doris never produces a file it cannot read.

Found while running Iceberg regression suites for #67366.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start in be/src/core/data_type_serde/parquet_timestamp.h, especially validate_parquet_timestamp_micros and the MIN_DORIS_TIMESTAMP_MICROS check. Reproduce with the issue's dt0 DATETIME(6) SQL and Parquet OUTFILE/local() queries, then inspect the relevant Parquet timestamp read tests. Done means Doris can round-trip its accepted year-zero DATETIME values without turning them into NULL, or consistently rejects them before writing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.