A Hive table fails to load if a custom schema is used.
- Langage dominant
- Scala
- Étoiles
- 31
- Forks
- 4
- Merge moyen
- 1 j 10 min
- PR mergées (30 j)
- 4
Description
## Describe the bug
Originally, this happened when decimal correction is used with Hive, and there are columns having decimal(38,18) types. Pramen tries to 'correct' the schema by applying a custom schema type.
```
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number1' (prec=38, scale=18) to decimal(38, 18)
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number2' (prec=38, scale=18) to decimal(38, 18)
JDBC connection error for jdbc:hive2://example.com:10000;AuthMech=1; No connection attempts left.
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input '.' expecting {'SELECT', ...
== SQL ==
tbl.number1 decimal(38, 18), tbl.number2 decimal(38, 18)
---^^^
Collapse
```
## Code and/or configuration snippet that caused the issue
```hocon
correct.decimals.in.schema = true
correct.decimals.fix.precision = true
```
## Expected behavior
- [X] Pramen shoule not 'correct' decimal(38,18) since precision and scale are within the range.
- [x] Investigate if custom schema support for Hive can be fixed since it is needed in case the data type does need correction.
## Context
- Pramen/pramen-py version: 1.8.5
- Spark version: 2.4.4
- Scala/Python version: 2.11
- Operating system: Linux
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.