A Hive table fails to load if a custom schema is used.
- Ngôn ngữ chính
- Scala
- Star
- 31
- Fork
- 4
- Merge trung bình
- 1 ngày 10 phút
- Pull request đã merge (30 ngày)
- 4
Mô tả
## Describe the bug
Originally, this happened when decimal correction is used with Hive, and there are columns having decimal(38,18) types. Pramen tries to 'correct' the schema by applying a custom schema type.
```
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number1' (prec=38, scale=18) to decimal(38, 18)
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number2' (prec=38, scale=18) to decimal(38, 18)
JDBC connection error for jdbc:hive2://example.com:10000;AuthMech=1; No connection attempts left.
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input '.' expecting {'SELECT', ...
== SQL ==
tbl.number1 decimal(38, 18), tbl.number2 decimal(38, 18)
---^^^
Collapse
```
## Code and/or configuration snippet that caused the issue
```hocon
correct.decimals.in.schema = true
correct.decimals.fix.precision = true
```
## Expected behavior
- [X] Pramen shoule not 'correct' decimal(38,18) since precision and scale are within the range.
- [x] Investigate if custom schema support for Hive can be fixed since it is needed in case the data type does need correction.
## Context
- Pramen/pramen-py version: 1.8.5
- Spark version: 2.4.4
- Scala/Python version: 2.11
- Operating system: Linux
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.