[Bug] Year-zero DATETIME survives the Parquet write but is rejected on read (MIN_DORIS_TIMESTAMP_MICROS stops at 0001-01-01)
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
master (`ddbaaab1388`)
### What's Wrong?
Doris writes year-zero `DATETIME` values to Parquet correctly, but cannot read its own file back — the values come back `NULL`.
`be/src/core/data_type_serde/parquet_timestamp.h` gates every Parquet timestamp read at year 1:
```cpp
inline constexpr int64_t MIN_DORIS_TIMESTAMP_MICROS = -62135596800000000LL; // 0001-01-01
inline constexpr int64_t MAX_DORIS_TIMESTAMP_MICROS = 253402300799999999LL; // 9999-12-31
inline Status validate_parquet_timestamp_micros(int64_t timestamp_micros) {
if (timestamp_micros < MIN_DORIS_TIMESTAMP_MICROS || timestamp_micros > MAX_DORIS_TIMESTAMP_MICROS) {
return Status::DataQualityError(
"Parquet timestamp is outside the Doris 0001-9999 range: micros={}", timestamp_micros);
}
return Status::OK();
}
```
But the DATETIME type itself starts at `0000-01-01`, whose micros value is `-62167219200000000` — below that bound. So the reader is stricter than the type it materialises into.
### What You Expected?
A `DATETIME` value that Doris accepts, stores and exports should read back from Doris's own Parquet file.
### How to Reproduce?
```sql
CREATE TABLE dt0 (id INT, ts DATETIME(6)) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1');
INSERT INTO dt0 VALUES (1,'0000-01-01 12:34:56'),(2,'0000-03-01 00:00:00'),
(3,'1969-12-31 23:59:59'),(4,'2024-01-01 12:00:00');
-- the type stores them fine
SELECT id, CAST(ts AS STRING) FROM dt0 ORDER BY id;
-- 1 0000-01-01 12:34:56.000000
-- 2 0000-03-01 00:00:00.000000
-- 3 1969-12-31 23:59:59.000000
-- 4 2024-01-01 12:00:00.000000
SELECT * FROM dt0 ORDER BY id INTO OUTFILE 'file:///tmp/dtz/p_' FORMAT AS PARQUET;
```
Read the file back with the `local()` TVF:
```
id ts_read
1 NULL <-- lost
2 NULL <-- lost
3 1969-12-31 23:59:59.000000
4 2024-01-01 12:00:00.000000
```
Note `0000-03-01` is also lost, so this is a plain range check, not a calendar edge case.
### Anything Else?
**The write side is correct**: Spark reads the same year-zero timestamps out of a Doris-written Iceberg table without trouble (verified with `apache/spark:4.0.0` against the Iceberg REST catalog — Spark returned `0000-01-01 12:34:56` for the row Doris returned `NULL` for). So only Doris's reader rejects them.
The constant name and the error text both say `0001-9999`, so the bound may be deliberate. If so, the asymmetry is still worth resolving in one direction or the other:
- widen the reader to `0000-01-01` (`-62167219200000000`) to match the DATETIME range, or
- reject year-zero timestamps on the write path too, so Doris never produces a file it cannot read.
Found while running Iceberg regression suites for #67366.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start in be/src/core/data_type_serde/parquet_timestamp.h, especially validate_parquet_timestamp_micros and the MIN_DORIS_TIMESTAMP_MICROS check. Reproduce with the issue's dt0 DATETIME(6) SQL and Parquet OUTFILE/local() queries, then inspect the relevant Parquet timestamp read tests. Done means Doris can round-trip its accepted year-zero DATETIME values without turning them into NULL, or consistently rejects them before writing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100