apache / apache/datafusion-comet
Port or enable DateTimeUtilsSuite timestamp format tests in Comet
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
While working on https://github.com/apache/datafusion-comet/pull/3730, it was discovered that we are not testing various date time formats because the tests in Spark's DateTimeUtilsSuite are never being called.
The test suite is in `sql/catalyst` and does not extend either `SQLTestUtils` or `CometTestBase`. The tests in this subproject are not part of Comet CI.
Also, `IgnoreComet` is not available in this subproject and it may not be possible to disable the tests selectively.
The following tests are relevant -
| Test | Format coverage |
|---|---|
| `string to timestamp` | `YYYY-MM-DD`, `YYYY-MM-DDTHH:mm:ss`, space-vs-T separator, time-only (`HH:mm:ss`, `T18:12:15`), timezone suffixes (`Z`, `UTC`, `GMT±hh:mm`, `±hhmm`, `±h:m`, named tz like `Europe/Moscow`), milliseconds/microseconds, nanosecond truncation, whitespace/control char
permutations, ~20 invalid formats that must return `None` |
| `SPARK-35780: support full range of timestamp string` | Negative years (`-1969-…`), 5-digit years (`99999-…`), zero-padded years, `Long.MaxValue`/`Long.MinValue` boundary timestamps, integer overflow in individual fields |
| `SPARK-37326: stringToTimestampWithoutTimeZone with allowTimeZone` | `TIMESTAMP_NTZ` with `allowTimeZone=true/false` and a `+hh:mm` suffix |
| `SPARK-15379: special invalid date string` | Feb 29 on non-leap year, Apr 31 — must be rejected by both `toDate` and `stringToTimestamp` |
| `trailing characters while converting string to timestamp` | Trailing garbage after a valid ISO timestamp (e.g. `2019-10-31T10:59:23Z:::`) must return `None` |
| `SPARK-29012: special timestamp values` | Keyword strings: `Epoch`, `NOW`, `Yesterday`, `Today`, `tomorrow` (with optional trailing timezone) |
| `SPARK-35979: special timestamp without time zone values` | Same keyword strings for `TIMESTAMP_NTZ` via `convertSpecialTimestampNTZ` |
It might be easier to copy the format-related tests from DateTimeUtilsSuite into a new or existing Comet test class (e.g. CometDateTimeUtilsSuite), as opposed to trying to enable catalyst tests in Comet CI.
Contributor guide
Assessment
This issue has not been assessed yet.