51zero / 51zero/eel-sdk

Need an example of creating DDL for a Hive Parquet table with EEL

Abierto Apto para principiantes
#261 0 comentarios 0 reacciones 1 asignado Reclamado por @hannesmiller Ver en GitHub
documentation
Lenguaje dominante
Scala
Estrellas
147
Forks
32
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

- CSVSource to HiveSink
```scala
val schema = AvroSchemaFns.fromAvroSchema(new Schema.Parser().parse(new File("user.avsc")))
CsvSource(path)
.withSchema(schema)
.to(HiveSink("mydatabase", "myTable"))
```
- Table field: **fname**, **lname**, **age**, **salary**
- 2 partition keys of **country** and **city**

```scala
object EelCreateTableExample extends App {
val crateTableCommand = HiveDDL.showDDL(
tableName = "mydatabase.mytable",
partitions = Seq(
PartitionColumn("country", StringType),
PartitionColumn("city", StringType)
),
fields = Seq(
Field("fname", StringType),
Field("lname", StringType),
Field("age", IntType.Signed),
Field("salary", DecimalType(38, 5))
),
tableType = TableType.EXTERNAL_TABLE,
location = Some("hdfs://nameservice1/blah/mytable_location"),
serde = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
inputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
outputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
props = Map.empty,
tableComment = Some("my lovely table"),
ifNotExists = true
)
println(crateTableCommand)
}
```

- Ouput:
```
CREATE EXTERNAL TABLE IF NOT EXISTS `mydatabase.mytable` (
`fname` string,
`lname` string,
`age` int,
`salary` decimal(38,5))
PARTITIONED BY (
`country` string,
`city` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'hdfs://nameservice1/blah/mytable_location'
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

The example is already provided in the issue body. Look for the HiveDDL class in the codebase to understand how showDDL is implemented. Verify the output matches the expected CREATE EXTERNAL TABLE statement. Add this example to the project's documentation, likely in a examples or docs directory. Run the example to ensure it compiles and prints the correct DDL.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
hadoop, scala
Área
data-engineering, databases
Tipo de issue
Documentación
Dificultad
2/5
Tiempo estimado
1-3 horas
Estado de actividad
Estancado
Claridad
Bien especificado
Aptitud para principiantes
75/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.