AbsaOSS / AbsaOSS/spark-data-standardization

Validation of timestamp with ISO8601 pattern with microseconds fails

Offen
#68 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Scala
Sterne
7
Forks
2
Ø Merge
1 T. 18 Std.
Gemergte PRs (30 T.)
4

Beschreibung

## Describe the bug
The following timestamp pattern fails to be validated: `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. This is a pattern defined by ISO8601. However, it does validate when removing either the `'T'` or the fractional second part (see test-cases below)

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Add the following test to `SchemaValidationSuite` and execute it
```
test("Timestamp with fractional seconds should be validated") {
val schema = StructType(
Array(
StructField("id", LongType),
StructField("name", StringType),
StructField("timestamp_without_fraction_ok", TimestampType, nullable = false, Metadata.fromJson(""" { "pattern": "yyyy-MM-dd'T'HH:mm:ssXXX", "default": "2025-01-08T13:07:39+00:00" } """.stripMargin)),
StructField("timestamp_with_fraction_without_T_ok", TimestampType, nullable = false, Metadata.fromJson(""" { "pattern": "yyyy-MM-dd HH:mm:ss.SSSXXX", "default": "2025-01-08 13:07:39.490+00:00" } """.stripMargin)),
StructField("timestamp_with_fraction_and_T_nok", TimestampType, nullable = false, Metadata.fromJson(""" { "pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSXXX", "default": "2025-01-08T13:07:39.490+00:00" } """.stripMargin))
)
)

val failures = SchemaValidator.validateSchema(schema)(CommonTypeDefaults)
if (failures.nonEmpty) {
logger.error("Validation errors:")
logger.error(failures.mkString("\n"))
}

assert(failures.isEmpty)
}
```
2. See the test failing with `Unparseable date: "2025-01-08T13:07:39.490:00`

## Expected behavior
The pattern with `'T'` and fractional seconds should be successfully validated.

## Use-case
Only for internal development / testing

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.