AbsaOSS / AbsaOSS/spark-data-standardization

Standardization can fail with `Task not serializable` when casting to date is requested

Ouverte
#57 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
bug
Langage dominant
Scala
Étoiles
7
Forks
2
Merge moyen
1 j 18 h
PR mergées (30 j)
4

Description

## Describe the bug
When conversion from an integer to a date in format of `yyyyMM` (possibly `YYYYMM`), Standardization had thrown a big error message starting from:
> Job aborted. (Job aborted due to stage failure: Task not serializable: = java.io.NotSerializableException: za.co.absa.standardization.config.Default= StandardizationConfig$ Serialization stack: - object not serializable (class: za.co.absa.standardization.config.Defaul= tStandardizationConfig$, value: za.co.absa.standardization.config.DefaultSt= andardizationConfig$@3ba3e058) - element of array (index: 3) - array (class [Ljava.lang.Object;, size 5) - field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, typ= e: class [Ljava.lang.Object;) - object (class java.lang.invoke.SerializedLambda, SerializedLambda[captur= ingClass=3Dclass za.co.absa.standardization.udf.UDFBuilder$, functionalInte= rfaceMethod=3Dscala/Function1.apply:(Ljava/lang/Object;)Ljava/lang/Object;,= implementation=3DinvokeStatic za/co/absa/standardization/udf/UDFBuilder$.$= anonfun$stringUdfViaNumericParser$1:(Lza/co/absa/standardization/types/pars= ers/NumericParser;ZLjava/lang/String;Lza/co/absa/standardization/config/Sta= ndardizationConfig;Lscala/Option;Ljava/lang/String;)Lza/co/absa/standardiza= tion/udf/UDFResult;, instantiatedMethodType=3D(Ljava/lang/String;)Lza/co/ab= sa/standardization/udf/UDFResult;, numCaptured=3D5]) - writeReplace data (class: java.lang.invoke.SerializedLambda) - object (class za.co.absa.standardization.udf.UDFBuilder$$$Lambda$5214/36= 0225670,

## To Reproduce
Steps to reproduce the behavior OR commands run:
1. Go to '...'
2. Use value '...'
3. Run using
4. See error

Try also to have an integer type with metadata:
```json
{"name": "MY_DATE", "type": "integer", "metadata": {"pattern": "yyyyMM", "timezone": "Africa/Johannesburg"}, "nullable": true}
```

## Expected behavior
- The standardization should have data errors if incorrect date format is passed, not failing the pipeline.
- If the above is not an option, at least, the error message should say which coulmn or condition have caused the error.

## Screenshots
If applicable, add screenshots to help explain your problem.

## Desktop (please complete the following information):
- OS: [e.g. iOS]
- Versions of libraries (Spark, Scala, ...)
- Version [e.g. 22]

## Additional context
Add any other context about the problem here.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.