AbsaOSS / AbsaOSS/pramen

A Hive table fails to load if a custom schema is used.

Open
#398 0 comments 0 reactions 0 assignees View on GitHub
bug DS Pramen-Scala
Dominant language
Scala
Stars
31
Forks
4
Avg merge
1d 10m
Merged PRs (30d)
4

Description

## Describe the bug
Originally, this happened when decimal correction is used with Hive, and there are columns having decimal(38,18) types. Pramen tries to 'correct' the schema by applying a custom schema type.

```
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number1' (prec=38, scale=18) to decimal(38, 18)
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number2' (prec=38, scale=18) to decimal(38, 18)

JDBC connection error for jdbc:hive2://example.com:10000;AuthMech=1; No connection attempts left.
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input '.' expecting {'SELECT', ...
== SQL ==
tbl.number1 decimal(38, 18), tbl.number2 decimal(38, 18)
---^^^
Collapse

```

## Code and/or configuration snippet that caused the issue

```hocon
correct.decimals.in.schema = true
correct.decimals.fix.precision = true
```

## Expected behavior
- [X] Pramen shoule not 'correct' decimal(38,18) since precision and scale are within the range.
- [x] Investigate if custom schema support for Hive can be fixed since it is needed in case the data type does need correction.

## Context
- Pramen/pramen-py version: 1.8.5
- Spark version: 2.4.4
- Scala/Python version: 2.11
- Operating system: Linux

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.