4paradigm / 4paradigm/OpenMLDB

Fail to get table constrains of Hive tables with Spark API

Offen
#3,072 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @tobegit3hub Auf GitHub ansehen
batch-engine bug
Vorherrschende Sprache
C++
Sterne
1.7k
Forks
331
Ø Merge
12 T. 12 Std.
Gemergte PRs (30 T.)
1

Beschreibung

Now we can not get table constrains like `not null` with Spark API. It may be the bug of Spark.

Here is the process to re-produce the issue.

We can create the table with constrains with hive.

```
create table notnull_col_table2 (col1 int not null, col2 int);
```

But we may not get the not null constrains with Spark API.

```
def main(argv: Array[String]): Unit = {

val spark = SparkSession.builder()
.master("local")
.config("spark.hadoop.hive.metastore.uris", "thrift://localhost:9083")
.config("spark.sql.catalogImplementation", "hive")
.getOrCreate()

val df = spark.table("db1.notnull_col_table2")

val schema = df.schema
println(schema) # StructType(StructField(col1,IntegerType,true),StructField(col2,IntegerType,true))

val col1_schema = schema.fields(0)
println(col1_schema.nullable) # true
}
```

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The issue is about Spark API not retrieving NOT NULL constraints from Hive tables. Start by examining how Spark's Hive integration fetches table metadata, particularly in the Spark SQL Hive client code. Look for where column nullability is determined. The test involves creating a Hive table with a NOT NULL column and checking the schema via spark.table. 'Done' means the nullable field in the schema correctly shows false for the constrained column.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
scala, spark
Bereich
data-engineering, databases
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.