AbsaOSS / AbsaOSS/spark-data-standardization

Validation of timestamp with ISO8601 pattern with microseconds fails

Abierto
#68 1 comentario 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Scala
Estrellas
7
Forks
2
Merge medio
1 d 18 h
PR fusionados (30 d)
4

Descripción

## Describe the bug
The following timestamp pattern fails to be validated: `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. This is a pattern defined by ISO8601. However, it does validate when removing either the `'T'` or the fractional second part (see test-cases below)

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Add the following test to `SchemaValidationSuite` and execute it
```
test("Timestamp with fractional seconds should be validated") {
val schema = StructType(
Array(
StructField("id", LongType),
StructField("name", StringType),
StructField("timestamp_without_fraction_ok", TimestampType, nullable = false, Metadata.fromJson(""" { "pattern": "yyyy-MM-dd'T'HH:mm:ssXXX", "default": "2025-01-08T13:07:39+00:00" } """.stripMargin)),
StructField("timestamp_with_fraction_without_T_ok", TimestampType, nullable = false, Metadata.fromJson(""" { "pattern": "yyyy-MM-dd HH:mm:ss.SSSXXX", "default": "2025-01-08 13:07:39.490+00:00" } """.stripMargin)),
StructField("timestamp_with_fraction_and_T_nok", TimestampType, nullable = false, Metadata.fromJson(""" { "pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSXXX", "default": "2025-01-08T13:07:39.490+00:00" } """.stripMargin))
)
)

val failures = SchemaValidator.validateSchema(schema)(CommonTypeDefaults)
if (failures.nonEmpty) {
logger.error("Validation errors:")
logger.error(failures.mkString("\n"))
}

assert(failures.isEmpty)
}
```
2. See the test failing with `Unparseable date: "2025-01-08T13:07:39.490:00`

## Expected behavior
The pattern with `'T'` and fractional seconds should be successfully validated.

## Use-case
Only for internal development / testing

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.