A Hive table fails to load if a custom schema is used.
- 主要言語
- Scala
- スター
- 31
- フォーク
- 4
- 平均マージ
- 1日 10分
- マージ済み PR(30日)
- 4
説明
## Describe the bug
Originally, this happened when decimal correction is used with Hive, and there are columns having decimal(38,18) types. Pramen tries to 'correct' the schema by applying a custom schema type.
```
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number1' (prec=38, scale=18) to decimal(38, 18)
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number2' (prec=38, scale=18) to decimal(38, 18)
JDBC connection error for jdbc:hive2://example.com:10000;AuthMech=1; No connection attempts left.
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input '.' expecting {'SELECT', ...
== SQL ==
tbl.number1 decimal(38, 18), tbl.number2 decimal(38, 18)
---^^^
Collapse
```
## Code and/or configuration snippet that caused the issue
```hocon
correct.decimals.in.schema = true
correct.decimals.fix.precision = true
```
## Expected behavior
- [X] Pramen shoule not 'correct' decimal(38,18) since precision and scale are within the range.
- [x] Investigate if custom schema support for Hive can be fixed since it is needed in case the data type does need correction.
## Context
- Pramen/pramen-py version: 1.8.5
- Spark version: 2.4.4
- Scala/Python version: 2.11
- Operating system: Linux
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。