51zero / 51zero/eel-sdk

Need an example of creating DDL for a Hive Parquet table with EEL

Ouverte Adaptée aux débutants
#261 0 commentaires 0 réactions 1 personne assignée Réclamée par @hannesmiller Voir sur GitHub
documentation
Langage dominant
Scala
Étoiles
147
Forks
32
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

- CSVSource to HiveSink
```scala
val schema = AvroSchemaFns.fromAvroSchema(new Schema.Parser().parse(new File("user.avsc")))
CsvSource(path)
.withSchema(schema)
.to(HiveSink("mydatabase", "myTable"))
```
- Table field: **fname**, **lname**, **age**, **salary**
- 2 partition keys of **country** and **city**

```scala
object EelCreateTableExample extends App {
val crateTableCommand = HiveDDL.showDDL(
tableName = "mydatabase.mytable",
partitions = Seq(
PartitionColumn("country", StringType),
PartitionColumn("city", StringType)
),
fields = Seq(
Field("fname", StringType),
Field("lname", StringType),
Field("age", IntType.Signed),
Field("salary", DecimalType(38, 5))
),
tableType = TableType.EXTERNAL_TABLE,
location = Some("hdfs://nameservice1/blah/mytable_location"),
serde = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
inputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
outputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
props = Map.empty,
tableComment = Some("my lovely table"),
ifNotExists = true
)
println(crateTableCommand)
}
```

- Ouput:
```
CREATE EXTERNAL TABLE IF NOT EXISTS `mydatabase.mytable` (
`fname` string,
`lname` string,
`age` int,
`salary` decimal(38,5))
PARTITIONED BY (
`country` string,
`city` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'hdfs://nameservice1/blah/mytable_location'
```

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

The example is already provided in the issue body. Look for the HiveDDL class in the codebase to understand how showDDL is implemented. Verify the output matches the expected CREATE EXTERNAL TABLE statement. Add this example to the project's documentation, likely in a examples or docs directory. Run the example to ensure it compiles and prints the correct DDL.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
hadoop, scala
Domaine
data-engineering, databases
Type d'issue
Documentation
Difficulté
2/5
Temps estimé
1-3 heures
Activité
À l'abandon
Clarté
Clairement spécifiée
Accessibilité débutants
75/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.